Dual-Screening Agreement Calculator: Cohen's Kappa and Conflict List for Systematic Reviews
Two reviewers screened the same records? Paste or upload both sets of decisions (include, exclude, maybe) and get Cohen’s kappa with its 95% confidence interval, percentage agreement, PABAK, Gwet’s AC1, a cross-table and a list of every conflict to discuss, ready to download. Works for title/abstract and full-text screening, and for risk-of-bias or data-extraction checks.
Lines are matched by position. Include: include, yes, y, 1, in, I. Exclude: exclude, no, n, 0, out, E. Maybe: maybe, unsure, ?, M. Empty lines in the ID box are allowed.
- PriceFree, no login or sign-up needed
- Your dataProcessed in your browser, never uploaded
- Works onPhone, tablet or computer, any modern browser
- Made byDr. Muhammad Aammar Tufail, Codanics
- Published17 November 2026
- Last updated16 November 2026
📚 How to cite this tool
This tool is free. If it helped your thesis, paper, report or assignment, please cite it. It keeps the tools free and helps other students find them.
Tufail, M. A. (2026). Dual-screening agreement calculator: Cohen's kappa and conflict list for systematic reviews [Online tool]. Codanics. Retrieved 16 November 2026, from https://codanics.com/tools/screening-agreement-kappa-calculator/
In-text citation: (Tufail, 2026) · Narrative: Tufail (2026)
The access date is today’s date, filled in automatically. Author: Muhammad Aammar Tufail. Published 17 November 2026.
How to use the dual-screening agreement calculator
- Paste the record IDs or titles and each reviewer’s decisions in the three boxes (one per line, in the same order), or use Paste from Excel / Upload CSV for an export from Rayyan, Covidence or a spreadsheet.
- If you only have counts, open the Agreement table tab and type the 2×2 (or 3×3) table.
- Decide how to treat ‘maybe’: as include (the usual rule at title/abstract stage, because maybes go to full text) or as a separate category, optionally with weighted kappa.
- Click Calculate agreement to get kappa with 95% CI, percent agreement, PABAK, Gwet’s AC1, the cross-table and the conflict list.
- Download the conflict list or the full resolution sheet to resolve disagreements, and copy the sentence into your methods section.
Why report screening agreement?
PRISMA 2020 and the Cochrane Handbook recommend that at least two reviewers screen records independently, and many journals ask how well they agreed. Cohen’s kappa corrects the raw percentage agreement for the agreement expected by chance. Because most records are excluded at the title and abstract stage, raw agreement is almost always high; kappa tells you whether the reviewers really agree on the few records that matter.
Formulas
| κ | Landis & Koch (1977) |
|---|---|
| < 0 | Poor |
| 0.00–0.20 | Slight |
| 0.21–0.40 | Fair |
| 0.41–0.60 | Moderate |
| 0.61–0.80 | Substantial |
| 0.81–1.00 | Almost perfect |
Worked example and validation
The example has 50 simulated abstracts. With ‘maybe’ counted as include, the reviewers agree on 44 of 50 records (88.0%); κ = 0.671 (95% CI 0.428 to 0.914), substantial agreement, PABAK = 0.76 and Gwet’s AC1 = 0.811. The tool lists the 6 conflicts for discussion. Keeping ‘maybe’ as a third category gives κ = 0.576 (unweighted), 0.627 (linear weights) and 0.655 (quadratic weights). We checked kappa, the z statistic and the p-value against irr::kappa2() (including the linear and quadratic weighted versions), and the standard error against vcd::Kappa() in R; all agree to at least 6 significant digits.
Frequently asked questions
What is a good kappa for title and abstract screening?
Most reviews aim for at least 0.6 (substantial agreement). Values above 0.8 are excellent. If kappa is low, discuss the disagreements, clarify the eligibility criteria and pilot again.
Why is kappa low when agreement is 95%?
When almost all records are excluded, chance agreement is very high, so kappa can be low even though the reviewers rarely disagree. This is the kappa paradox. Report PABAK or Gwet’s AC1 as well, which are less affected by prevalence.
How should I handle 'maybe' decisions?
At the title/abstract stage, a ‘maybe’ usually means the record goes to full-text review, so treating it as include is common. You can also keep it as a third category; then weighted kappa gives partial credit for include-vs-maybe disagreements.
Can I use this for full-text screening or risk-of-bias judgements?
Yes. Any two sets of categorical decisions on the same items work: full-text decisions, risk-of-bias judgements, or extracted categories.
Does this work with Rayyan or Covidence exports?
Yes. Export the decisions of both reviewers to CSV or Excel, then upload the file and choose which columns hold the record ID and each reviewer’s decision.
Is my data uploaded?
No. Everything runs in your browser.
Do I need to log in or create an account to use the dual-screening agreement calculator?
No. There is no login, no sign-up and no email required. Open the page and start using it straight away, for free, on any device.
How do I cite this tool in my thesis or paper?
Use the citation box on this page. In APA 7th edition: Tufail, M. A. (2026). Dual-screening agreement calculator: Cohen's kappa and conflict list for systematic reviews [Online tool]. Codanics. Retrieved [date you used it], from https://codanics.com/tools/screening-agreement-kappa-calculator/. BibTeX, RIS (EndNote, Mendeley, Zotero), MLA, Harvard, Chicago and Vancouver formats are also available.
Free guides and courses from Codanics
More free tools
See all free Codanics tools → · Codanics on YouTube · WhatsApp channel

