Weighted Kappa Calculator (Quadratic & Linear)

Weighted kappa measures how much two raters agree on an ordinalscale — grades 1–5, severity Low/Medium/High, or a model's predicted class versus the true label — while giving partial credit for near misses. It follows Cohen's 1968 definition and reproduces scikit-learn's cohen_kappa_score to the digit.

1. Arrange your data as a k×k count matrix O, where O_ij is the number of items rater A put in category i and rater B in category j. N is the grand total.
2. Compute the row totals R_i and column totals C_j, then the expected matrix E_ij = R_i·C_j / N. This is what the counts would look like if the two raters were independent.
3. Build the disagreement weight matrix, which is zero on the diagonal and grows with the distance between categories:
   • Linear: wᵢⱼ = |i − j| / (k − 1)
   • Quadratic: wᵢⱼ = (i − j)² / (k − 1)²
4. Take the weighted sums of the observed and expected matrices, then apply Cohen's formula:

   κw = 1 − (Σ wᵢⱼ Oᵢⱼ) / (Σ wᵢⱼ Eᵢⱼ)

   A κ_w of 1 means perfect agreement, 0 means chance-level, and negatives mean worse than chance.

The (k − 1) normalisation in the weights cancels in the ratio, so this form is identical to the unnormalised weights scikit-learn uses. The unweighted κ shown alongside is the same calculation with binary weights (0 on the diagonal, 1 everywhere else), which reduces to the familiar κ = (p_o − p_e) / (1 − p_e). Because weighting hands out partial credit for near-misses, you will normally see quadratic κ ≥ linear κ ≥ unweighted κ on the same matrix — a quick sanity check that the numbers are right.

To label the strength of agreement, the tool uses the Landis & Koch (1977) bands: below 0 is poor (worse than chance), up to 0.20 is slight, 0.40 fair, 0.60 moderate, 0.80 substantial, and above that almost perfect. Every result is verified internally by recomputing κ_w through an equivalent agreement-weight form; the two paths must produce the same number.

All three examples use the same 3×3 matrix (k = 3, N = 50), rows = Rater A, columns = Rater B: [[10, 5, 1], [3, 12, 4], [1, 6, 8]]. Expected diagonal: E₀₀ = 4.48, E₁₁ = 8.74, E₂₂ = 3.90.

• Cohen, J. (1968) — Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit, Psychological Bulletin 70(4)
• Landis, J.R. & Koch, G.G. (1977) — The measurement of observer agreement for categorical data, Biometrics 33(1)
• scikit-learn — cohen_kappa_score reference (weights: linear / quadratic)

Formulas last cross-checked against these sources and against scikit-learn on 2026-07-13. The sample 3×3 matrix reproduces κw = 0.5514 (quadratic), 0.4665 (linear), and 0.3917 (unweighted).