Regression & multivariate

Inter-Rater Reliability Calculator: Cohen's Kappa, Weighted Kappa, Fleiss' Kappa and ICC

Check how well two or more raters, reviewers or instruments agree. Paste the ratings and get Cohen’s kappa (unweighted, linear or quadratic weights) with a confidence interval, Fleiss’ kappa for three or more raters, or the intraclass correlation coefficient (ICC) in all six forms, with interpretation and a sentence for your methods. For systematic reviews, the screening conflicts are listed for you to resolve.

✅ No login needed100% freeRuns in your browserYour data stays on your device
Kappa and ICC Calculator✅ Free · no login📚 Cite this tool
Your data never leaves your PC. Your data is analysed on your own computer or phone, even when you upload a CSV or Excel file. Nothing is uploaded, stored or seen by Codanics, and no login is needed.

  • PriceFree, no login or sign-up needed
  • Your dataProcessed in your browser, never uploaded
  • Works onPhone, tablet or computer, any modern browser
  • Made byDr. Muhammad Aammar Tufail, Codanics
  • Published17 November 2026
  • Last updated16 November 2026

📚 How to cite this tool

This tool is free. If it helped your thesis, paper, report or assignment, please cite it. It keeps the tools free and helps other students find them.

Citation · APA 7th edition

Tufail, M. A. (2026). Kappa and ICC calculator: inter-rater reliability [Online tool]. Codanics. Retrieved 16 November 2026, from https://codanics.com/tools/kappa-icc-calculator/

In-text citation: (Tufail, 2026) · Narrative: Tufail (2026)

The access date is today’s date, filled in automatically. Author: Muhammad Aammar Tufail. Published 17 November 2026.

🚀 Follow Codanics channel on WhatsApp for more updatesNew free tools, scholarships, PhD positions and research tips, straight to your phone. Free lessons on YouTube too.

How to use the kappa and icc calculator

  1. Choose the tab: Two raters for Cohen’s kappa (categories such as include/exclude, present/absent or a 1–5 grade), 3+ raters for Fleiss’ kappa, or Continuous ratings for the ICC (scores, measurements, lengths).
  2. Paste the ratings with a header row, or use Paste from Excel / Upload CSV / Excel. Each row is one item or subject, each column one rater. Every tab starts with an example: two reviewers screening 50 abstracts, six psychiatrists diagnosing 30 patients (Fleiss, 1971) and four judges rating six targets (Shrout & Fleiss, 1979).
  3. For ordinal categories, choose linear or quadratic weights and check the category order. For the ICC, pick the model, type and unit that match your design (see the guide below).
  4. Read the coefficient, its confidence interval and the interpretation, copy the results sentence and tables into Word, and download the list of disagreements to resolve screening conflicts.

Which reliability statistic should I use?

Your dataRatersStatistic
Nominal categories (include/exclude, diagnosis)2Cohen’s kappa
Ordinal categories (mild/moderate/severe, 1–5 scores)2Weighted kappa (quadratic weights are the most common)
Nominal categories3 or moreFleiss’ kappa
Continuous measurements or scores2 or moreIntraclass correlation coefficient (ICC)
Agreement between two measurement methods2 methodsBland–Altman analysis plus the ICC
Internal consistency of questionnaire items–Cronbach’s alpha

Cohen’s kappa

Percentage agreement overstates reliability because raters agree sometimes by chance. Kappa corrects for this:

κ = (pₒ − pₑ) / (1 − pₑ) pₒ = observed proportion of agreement, pₑ = Σ (row proportion × column proportion) Weighted κ: pₒ and pₑ use weights wᵢⱼ = 1 − |i − j|/(k − 1) (linear) or 1 − (i − j)²/(k − 1)² (quadratic) SE for the CI: Fleiss, Cohen & Everitt (1969); z = κ / SE₀ (SE under κ = 0) PABAK = 2pₒ − 1

Kappa, z and p match irr::kappa2(); the confidence interval matches psych::cohen.kappa(). Landis and Koch (1977) describe κ of 0.01–0.20 as slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial and 0.81–1.00 almost perfect agreement.

Kappa in systematic reviews

Cochrane and PRISMA 2020 recommend that two reviewers screen titles, abstracts and full texts independently. Report the agreement (for example “κ = 0.78”) and how disagreements were resolved. Paste both reviewers’ decisions, and this tool lists every conflict for discussion or a third reviewer. Build your flow diagram with the PRISMA flow diagram generator.

Fleiss’ kappa

Pᵢ = [Σⱼ nᵢⱼ² − m] / [m(m − 1)] (agreement on subject i, m raters) P̄ = mean of Pᵢ, P̄ₑ = Σⱼ pⱼ², κ = (P̄ − P̄ₑ) / (1 − P̄ₑ)

Fleiss’ kappa allows different raters for different subjects, as long as each subject is rated by the same number of raters. Results, including the kappa for each category, match irr::kappam.fleiss(detail = TRUE).

Intraclass correlation coefficient (ICC)

Choose the form from your design (Koo & Li, 2016):

  • Model: two-way when the same raters rate every subject (the usual case); one-way when each subject is rated by a different, random set of raters.
  • Type: absolute agreement when the raters’ actual values must match (for example two devices); consistency when only the ranking matters and a constant offset between raters is acceptable.
  • Unit: single if in practice one rater’s score will be used; average if you will use the mean of all raters.
ICC(A,1) = (MSR − MSE) / [MSR + (k − 1)MSE + k(MSC − MSE)/n] ICC(C,1) = (MSR − MSE) / [MSR + (k − 1)MSE] ICC(1) = (MSR − MSW) / [MSR + (k − 1)MSW] MSR = subjects, MSC = raters, MSE = residual, MSW = within-subject mean squares

All six ICCs, F tests and confidence intervals match irr::icc() and psych::ICC(). On the Shrout & Fleiss (1979) example, ICC(1,1) = 0.17, ICC(2,1) = 0.29 and ICC(3,1) = 0.71, as in the original paper.

How to report reliability

Two reviewers independently screened 50 abstracts. Agreement was moderate, Cohen’s κ = .58, 95% CI [.39, .78]; disagreements were resolved by discussion. Test–retest reliability of the scale was good, ICC(A,1) = .86, 95% CI [.74, .93], based on a two-way random-effects, absolute-agreement, single-rater model.

Plan reliability into your review with the free systematic review and meta-analysis guide.

Learn the statistics behind these numbers. Codanics has free statistics, R and Python lessons on YouTube and full courses for researchers.

Frequently asked questions

What is a good kappa value?

By Landis and Koch’s scale, 0.61–0.80 is substantial and above 0.80 almost perfect agreement. Many systematic reviews aim for kappa of at least 0.6 at the abstract stage before splitting the screening. Always report the confidence interval and percentage agreement too.

Why is my kappa low when percentage agreement is high?

When one category is very common (for example 90% of abstracts are excluded), agreement by chance is already high, so kappa can be low even with 92% agreement. This is the ‘kappa paradox’. Report the PABAK and the agreement per category alongside kappa.

Should I use linear or quadratic weights?

Use weights only for ordered categories. Quadratic weights penalise large disagreements much more than near misses and are the most common choice; quadratically weighted kappa is close to the ICC. Linear weights penalise all steps equally. Say which weights you used.

Which ICC should I report?

For most inter-rater studies where the same raters rate every subject and their actual values should agree, report ICC(2,1), also called ICC(A,1): two-way random effects, absolute agreement, single rater. Report the model, type and unit, as different ICC forms can give very different values.

Can I use kappa with more than two raters?

Cohen’s kappa is for exactly two raters. For three or more raters with categorical ratings, use Fleiss’ kappa (the second tab). For continuous ratings, use the ICC with any number of raters.

What happens to missing ratings?

Items or subjects with a missing rating from any selected rater are left out, and the number skipped is shown. Fill in or remove incomplete rows if many are skipped.

Is my data uploaded?

No. All calculations run in your browser. Pasted and uploaded ratings never leave your device.

Do I need to log in or create an account to use the kappa and icc calculator?

No. There is no login, no sign-up and no email required. Open the page and start using it straight away, for free, on any device.

How do I cite this tool in my thesis or paper?

Use the citation box on this page. In APA 7th edition: Tufail, M. A. (2026). Kappa and ICC calculator: inter-rater reliability [Online tool]. Codanics. Retrieved [date you used it], from https://codanics.com/tools/kappa-icc-calculator/. BibTeX, RIS (EndNote, Mendeley, Zotero), MLA, Harvard, Chicago and Vancouver formats are also available.

Free guides and courses from Codanics

More free tools

See all free Codanics tools → · Codanics on YouTube · WhatsApp channel

Made by Dr. Muhammad Aammar Tufail, Codanics. First published 17 November 2026, last updated 16 November 2026. Found a bug or have an idea for a new tool? Tell us on the WhatsApp channel or in the comments on YouTube. Results are for learning and planning; check important analyses with a statistician or your supervisor.