Inter-Rater Reliability Calculator: Cohen's Kappa, Weighted Kappa, Fleiss' Kappa and ICC
Check how well two or more raters, reviewers or instruments agree. Paste the ratings and get Cohen’s kappa (unweighted, linear or quadratic weights) with a confidence interval, Fleiss’ kappa for three or more raters, or the intraclass correlation coefficient (ICC) in all six forms, with interpretation and a sentence for your methods. For systematic reviews, the screening conflicts are listed for you to resolve.
Rater columns
Rater columns
- PriceFree, no login or sign-up needed
- Your dataProcessed in your browser, never uploaded
- Works onPhone, tablet or computer, any modern browser
- Made byDr. Muhammad Aammar Tufail, Codanics
- Published17 November 2026
- Last updated16 November 2026
📚 How to cite this tool
This tool is free. If it helped your thesis, paper, report or assignment, please cite it. It keeps the tools free and helps other students find them.
Tufail, M. A. (2026). Kappa and ICC calculator: inter-rater reliability [Online tool]. Codanics. Retrieved 16 November 2026, from https://codanics.com/tools/kappa-icc-calculator/
In-text citation: (Tufail, 2026) · Narrative: Tufail (2026)
The access date is today’s date, filled in automatically. Author: Muhammad Aammar Tufail. Published 17 November 2026.
How to use the kappa and icc calculator
- Choose the tab: Two raters for Cohen’s kappa (categories such as include/exclude, present/absent or a 1–5 grade), 3+ raters for Fleiss’ kappa, or Continuous ratings for the ICC (scores, measurements, lengths).
- Paste the ratings with a header row, or use Paste from Excel / Upload CSV / Excel. Each row is one item or subject, each column one rater. Every tab starts with an example: two reviewers screening 50 abstracts, six psychiatrists diagnosing 30 patients (Fleiss, 1971) and four judges rating six targets (Shrout & Fleiss, 1979).
- For ordinal categories, choose linear or quadratic weights and check the category order. For the ICC, pick the model, type and unit that match your design (see the guide below).
- Read the coefficient, its confidence interval and the interpretation, copy the results sentence and tables into Word, and download the list of disagreements to resolve screening conflicts.
Which reliability statistic should I use?
| Your data | Raters | Statistic |
|---|---|---|
| Nominal categories (include/exclude, diagnosis) | 2 | Cohen’s kappa |
| Ordinal categories (mild/moderate/severe, 1–5 scores) | 2 | Weighted kappa (quadratic weights are the most common) |
| Nominal categories | 3 or more | Fleiss’ kappa |
| Continuous measurements or scores | 2 or more | Intraclass correlation coefficient (ICC) |
| Agreement between two measurement methods | 2 methods | Bland–Altman analysis plus the ICC |
| Internal consistency of questionnaire items | – | Cronbach’s alpha |
Cohen’s kappa
Percentage agreement overstates reliability because raters agree sometimes by chance. Kappa corrects for this:
Kappa, z and p match irr::kappa2(); the confidence interval matches psych::cohen.kappa(). Landis and Koch (1977) describe κ of 0.01–0.20 as slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial and 0.81–1.00 almost perfect agreement.
Kappa in systematic reviews
Cochrane and PRISMA 2020 recommend that two reviewers screen titles, abstracts and full texts independently. Report the agreement (for example “κ = 0.78”) and how disagreements were resolved. Paste both reviewers’ decisions, and this tool lists every conflict for discussion or a third reviewer. Build your flow diagram with the PRISMA flow diagram generator.
Fleiss’ kappa
Fleiss’ kappa allows different raters for different subjects, as long as each subject is rated by the same number of raters. Results, including the kappa for each category, match irr::kappam.fleiss(detail = TRUE).
Intraclass correlation coefficient (ICC)
Choose the form from your design (Koo & Li, 2016):
- Model: two-way when the same raters rate every subject (the usual case); one-way when each subject is rated by a different, random set of raters.
- Type: absolute agreement when the raters’ actual values must match (for example two devices); consistency when only the ranking matters and a constant offset between raters is acceptable.
- Unit: single if in practice one rater’s score will be used; average if you will use the mean of all raters.
All six ICCs, F tests and confidence intervals match irr::icc() and psych::ICC(). On the Shrout & Fleiss (1979) example, ICC(1,1) = 0.17, ICC(2,1) = 0.29 and ICC(3,1) = 0.71, as in the original paper.
How to report reliability
Plan reliability into your review with the free systematic review and meta-analysis guide.
Frequently asked questions
What is a good kappa value?
By Landis and Koch’s scale, 0.61–0.80 is substantial and above 0.80 almost perfect agreement. Many systematic reviews aim for kappa of at least 0.6 at the abstract stage before splitting the screening. Always report the confidence interval and percentage agreement too.
Why is my kappa low when percentage agreement is high?
When one category is very common (for example 90% of abstracts are excluded), agreement by chance is already high, so kappa can be low even with 92% agreement. This is the ‘kappa paradox’. Report the PABAK and the agreement per category alongside kappa.
Should I use linear or quadratic weights?
Use weights only for ordered categories. Quadratic weights penalise large disagreements much more than near misses and are the most common choice; quadratically weighted kappa is close to the ICC. Linear weights penalise all steps equally. Say which weights you used.
Which ICC should I report?
For most inter-rater studies where the same raters rate every subject and their actual values should agree, report ICC(2,1), also called ICC(A,1): two-way random effects, absolute agreement, single rater. Report the model, type and unit, as different ICC forms can give very different values.
Can I use kappa with more than two raters?
Cohen’s kappa is for exactly two raters. For three or more raters with categorical ratings, use Fleiss’ kappa (the second tab). For continuous ratings, use the ICC with any number of raters.
What happens to missing ratings?
Items or subjects with a missing rating from any selected rater are left out, and the number skipped is shown. Fill in or remove incomplete rows if many are skipped.
Is my data uploaded?
No. All calculations run in your browser. Pasted and uploaded ratings never leave your device.
Do I need to log in or create an account to use the kappa and icc calculator?
No. There is no login, no sign-up and no email required. Open the page and start using it straight away, for free, on any device.
How do I cite this tool in my thesis or paper?
Use the citation box on this page. In APA 7th edition: Tufail, M. A. (2026). Kappa and ICC calculator: inter-rater reliability [Online tool]. Codanics. Retrieved [date you used it], from https://codanics.com/tools/kappa-icc-calculator/. BibTeX, RIS (EndNote, Mendeley, Zotero), MLA, Harvard, Chicago and Vancouver formats are also available.
Free guides and courses from Codanics
More free tools
See all free Codanics tools → · Codanics on YouTube · WhatsApp channel