Cohen's Kappa (Inter-Rater Agreement)

Every accuracy measure on this site assumes something is true. Kappa is what you compute when nothing is: two raters classify the same items, neither is the standard, and the question is how far they agree beyond what their own habits would have produced by chance.

Raw agreement is not the answer, because it is inflated by any imbalance in the categories. If two radiologists each call 95 % of films normal, they will agree on roughly 90 % of films by accident alone, and 90 % agreement is then no evidence of skill whatsoever. Jacob Cohen's 1960 statistic subtracts that floor and rescales what is left: $\kappa = (p_o - p_e)/(1 - p_e)$, where the numerator is the agreement above chance and the denominator is the agreement that was available above chance. The value is 1 for perfect agreement, 0 for chance-level agreement, and negative for systematic disagreement.

The interpretive bands in common use — Landis and Koch's 1977 labels of slight, fair, moderate, substantial and almost perfect — are worth naming and distrusting in the same breath. Landis and Koch offered them as arbitrary, and they have hardened into something read as though it were a measurement standard. What counts as adequate agreement depends on what the classification decides, and a kappa that is fine for sorting survey responses is not fine for deciding who gets a biopsy.

Two structural quirks deserve care. Kappa falls as prevalence becomes extreme even when raw agreement is high, because the chance term climbs — the well-documented kappa paradox, in which two readers agreeing on 95 % of films can post a kappa near zero. And plain kappa treats every disagreement as equally bad, which is wrong for ordered categories where mild-versus-moderate is a smaller error than normal-versus-severe; weighted kappa exists for that case. Kappa answers whether two observers can be trusted to produce the same answer. It says nothing about whether either of them is right.