Cohen's Kappa Calculator (Inter-Rater Reliability)

Cohen's kappa measures how much two raters agree when each independently sorts the same items into the same set of categories, and crucially, it corrects for the agreement you would expect from pure chance. You give the tool a k×k agreement matrix where nᵢⱼ is the number of items Rater A put in category i and Rater B put in category j. The diagonal holds the agreements; everything off the diagonal is a disagreement.

From the grand total N, the row marginals rᵢ and the column marginals cⱼ, the calculation follows Cohen (1960):

• Observed agreement: pₒ = (Σ nᵢᵢ) / N
• Chance agreement: pₑ = Σ (rᵢ / N)(cᵢ / N)
• Cohen's kappa: κ = (pₒ − pₑ) / (1 − pₑ)

κ = 1 is perfect agreement, κ = 0 is exactly what chance predicts, and κ < 0 means the raters disagree more than random labelling would. Because pₑ depends on the marginal totals, a lopsided task, where one category dominates, has a high chance agreement, which is why two raters can match on 80% of items yet earn only a modest kappa.

When the categories are ordered (say Low / Medium / High), a Medium-vs-High mix-up is a smaller error than Low-vs-High. Weighted kappa, from Cohen (1968), captures that with a weight matrix built from the category distance |i − j|: linear weights are wᵢⱼ = 1 − |i−j|/(k−1) and quadratic weights are wᵢⱼ = 1 − (i−j)²/(k−1)². The same formula then runs on the weighted proportions: κ_w = (pₒ(w) − pₑ(w)) / (1 − pₑ(w)). For a 2×2 table every weighting scheme collapses to plain kappa, since there is only one disagreement distance.

The 95% confidence interval uses Cohen's simplified large-sample standard error, SE = √(pₒ(1 − pₒ) / (N·(1 − pₑ)²)), giving κ ± 1.96·SE clamped to the valid −1…1 range. This normal approximation is dependable for N ≥ 30 and is flagged as indicative below that; the full asymptotic variance is given by Fleiss, Cohen & Everitt (1969). Finally the headline kappa is mapped to the Landis & Koch (1977) band, < 0.00 poor, 0.00–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect, for a one-line verdict. All arithmetic is exact and runs in your browser; nothing is uploaded.

• Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37–46: the original κ definition and standard error.
• Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement. Psychological Bulletin, 70(4), 213–220: linear/quadratic weighted kappa.
• Landis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159–174: the strength-of-agreement bands.
• Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72(5), 323–327: the full asymptotic variance.

Every formula on this page was cross-checked against these sources on 2026-06-10, and the unweighted result is verified against the direct one-line formula inside the tool. Your agreement matrix never leaves your browser.