Site icon Codanics

Screening Agreement Calculator: Cohen’s Kappa

Dual-Screening Agreement Calculator: free online tool by Codanics that runs in your browser
Systematic review & meta-analysis

Dual-Screening Agreement Calculator: Cohen's Kappa and Conflict List for Systematic Reviews

Two reviewers screened the same records? Paste or upload both sets of decisions (include, exclude, maybe) and get Cohen’s kappa with its 95% confidence interval, percentage agreement, PABAK, Gwet’s AC1, a cross-table and a list of every conflict to discuss, ready to download. Works for title/abstract and full-text screening, and for risk-of-bias or data-extraction checks.

✅ No login needed100% freeRuns in your browserYour data stays on your device
Dual-Screening Agreement Calculator✅ Free · no login📚 Cite this tool
Your data never leaves your PC. Your data is analysed on your own computer or phone, even when you upload a CSV or Excel file. Nothing is uploaded, stored or seen by Codanics, and no login is needed.

Lines are matched by position. Include: include, yes, y, 1, in, I. Exclude: exclude, no, n, 0, out, E. Maybe: maybe, unsure, ?, M. Empty lines in the ID box are allowed.

  • PriceFree, no login or sign-up needed
  • Your dataProcessed in your browser, never uploaded
  • Works onPhone, tablet or computer, any modern browser
  • Made byDr. Muhammad Aammar Tufail, Codanics
  • Published17 November 2026
  • Last updated16 November 2026

📚 How to cite this tool

This tool is free. If it helped your thesis, paper, report or assignment, please cite it. It keeps the tools free and helps other students find them.

Citation · APA 7th edition

Tufail, M. A. (2026). Dual-screening agreement calculator: Cohen's kappa and conflict list for systematic reviews [Online tool]. Codanics. Retrieved 16 November 2026, from https://codanics.com/tools/screening-agreement-kappa-calculator/

In-text citation: (Tufail, 2026) · Narrative: Tufail (2026)

The access date is today’s date, filled in automatically. Author: Muhammad Aammar Tufail. Published 17 November 2026.

🚀 Follow Codanics channel on WhatsApp for more updatesNew free tools, scholarships, PhD positions and research tips, straight to your phone. Free lessons on YouTube too.

How to use the dual-screening agreement calculator

  1. Paste the record IDs or titles and each reviewer’s decisions in the three boxes (one per line, in the same order), or use Paste from Excel / Upload CSV for an export from Rayyan, Covidence or a spreadsheet.
  2. If you only have counts, open the Agreement table tab and type the 2×2 (or 3×3) table.
  3. Decide how to treat ‘maybe’: as include (the usual rule at title/abstract stage, because maybes go to full text) or as a separate category, optionally with weighted kappa.
  4. Click Calculate agreement to get kappa with 95% CI, percent agreement, PABAK, Gwet’s AC1, the cross-table and the conflict list.
  5. Download the conflict list or the full resolution sheet to resolve disagreements, and copy the sentence into your methods section.

Why report screening agreement?

PRISMA 2020 and the Cochrane Handbook recommend that at least two reviewers screen records independently, and many journals ask how well they agreed. Cohen’s kappa corrects the raw percentage agreement for the agreement expected by chance. Because most records are excluded at the title and abstract stage, raw agreement is almost always high; kappa tells you whether the reviewers really agree on the few records that matter.

Formulas

Observed agreement pₒ = Σᵢ Σⱼ wᵢⱼ pᵢⱼ (wᵢⱼ = 1 on the diagonal, 0 elsewhere for unweighted kappa) Chance agreement pₑ = Σᵢ Σⱼ wᵢⱼ pᵢ. p.ⱼ Cohen’s kappa κ = (pₒ − pₑ) / (1 − pₑ) SE for the 95% CI Fleiss, Cohen & Everitt (1969) large-sample variance z-test z = κ / SE₀ (SE under H₀: κ = 0, as in R irr::kappa2) PABAK (q·pₒ − 1)/(q − 1) = 2pₒ − 1 for two categories Gwet’s AC1 (pₒ − pₑ*)/(1 − pₑ*), pₑ* = Σ πₖ(1 − πₖ)/(q − 1), πₖ = (pₖ. + p.ₖ)/2 Weighted kappa linear wᵢⱼ = 1 − |i − j|/(q − 1); quadratic wᵢⱼ = 1 − (i − j)²/(q − 1)²
κLandis & Koch (1977)
< 0Poor
0.00–0.20Slight
0.21–0.40Fair
0.41–0.60Moderate
0.61–0.80Substantial
0.81–1.00Almost perfect

Worked example and validation

The example has 50 simulated abstracts. With ‘maybe’ counted as include, the reviewers agree on 44 of 50 records (88.0%); κ = 0.671 (95% CI 0.428 to 0.914), substantial agreement, PABAK = 0.76 and Gwet’s AC1 = 0.811. The tool lists the 6 conflicts for discussion. Keeping ‘maybe’ as a third category gives κ = 0.576 (unweighted), 0.627 (linear weights) and 0.655 (quadratic weights). We checked kappa, the z statistic and the p-value against irr::kappa2() (including the linear and quadratic weighted versions), and the standard error against vcd::Kappa() in R; all agree to at least 6 significant digits.

Good practice: pilot the eligibility criteria on 50–100 records, calculate kappa, refine the criteria and repeat until agreement is at least substantial. Report your screening numbers in a PRISMA flow diagram.
Want to run a full systematic review and meta-analysis? Our two-month, self-paced course takes you from your research question to a publishable review, with R, Excel, Word and EndNote. It is taught in Urdu/Hindi by Dr. Aammar Tufail.

Frequently asked questions

What is a good kappa for title and abstract screening?

Most reviews aim for at least 0.6 (substantial agreement). Values above 0.8 are excellent. If kappa is low, discuss the disagreements, clarify the eligibility criteria and pilot again.

Why is kappa low when agreement is 95%?

When almost all records are excluded, chance agreement is very high, so kappa can be low even though the reviewers rarely disagree. This is the kappa paradox. Report PABAK or Gwet’s AC1 as well, which are less affected by prevalence.

How should I handle 'maybe' decisions?

At the title/abstract stage, a ‘maybe’ usually means the record goes to full-text review, so treating it as include is common. You can also keep it as a third category; then weighted kappa gives partial credit for include-vs-maybe disagreements.

Can I use this for full-text screening or risk-of-bias judgements?

Yes. Any two sets of categorical decisions on the same items work: full-text decisions, risk-of-bias judgements, or extracted categories.

Does this work with Rayyan or Covidence exports?

Yes. Export the decisions of both reviewers to CSV or Excel, then upload the file and choose which columns hold the record ID and each reviewer’s decision.

Is my data uploaded?

No. Everything runs in your browser.

Do I need to log in or create an account to use the dual-screening agreement calculator?

No. There is no login, no sign-up and no email required. Open the page and start using it straight away, for free, on any device.

How do I cite this tool in my thesis or paper?

Use the citation box on this page. In APA 7th edition: Tufail, M. A. (2026). Dual-screening agreement calculator: Cohen's kappa and conflict list for systematic reviews [Online tool]. Codanics. Retrieved [date you used it], from https://codanics.com/tools/screening-agreement-kappa-calculator/. BibTeX, RIS (EndNote, Mendeley, Zotero), MLA, Harvard, Chicago and Vancouver formats are also available.

Free guides and courses from Codanics

More free tools

See all free Codanics tools → · Codanics on YouTube · WhatsApp channel

Made by Dr. Muhammad Aammar Tufail, Codanics. First published 17 November 2026, last updated 16 November 2026. Found a bug or have an idea for a new tool? Tell us on the WhatsApp channel or in the comments on YouTube. Results are for learning and planning; check important analyses with a statistician or your supervisor.
Exit mobile version