ROC Curve & AUC Calculator

The ROC curveplots a classifier's true positive rate against its false positive rate as the decision threshold moves, and the AUCis the single number underneath it. The method here follows scikit-learn's roc_curve / roc_auc_score and the definitions in Fawcett's 2006 ROC primer.

With P positive and N negative samples, each threshold yields a confusion matrix and a point:

TPR = TP / P   FPR = FP / N

1. Sortthe samples by predicted score, descending. If you pick “lower score = positive”, the scores are negated first so the sweep can always treat a higher value as more positive.
2. Sweep the threshold through every distinct score. Classify a sample positive when its score meets the threshold, and record the running TPR and FPR — one ROC point per distinct score, with (0,0) prepended and (1,1) appended.
3. Integrate the area under those points with the trapezoidal rule:

   AUC = Σ (FPRᵢ − FPRᵢ₋₁)·(TPRᵢ + TPRᵢ₋₁)/2

4. Cross-checkwith the Mann–Whitney / Wilcoxon rank form (Hanley & McNeil 1982). Rank every score ascending — averaging ranks for ties — and with R⁺ the sum of the positive ranks:

   AUC = (R⁺ − P(P+1)/2) / (P·N)

   This equals the probability that a random positive outranks a random negative (ties counted as half) and must match the trapezoidal value — the tool asserts the two agree to floating-point precision.

From the AUC the tool reports the Gini coefficient = 2·AUC − 1 and a plain-English discrimination band. For each threshold it also computes Youden's J = TPR − FPR; the threshold with the highest J — the point on the curve furthest from the no-skill diagonal — is flagged as a common operating-point choice. Inputs with only one class are rejected, because the AUC is then undefined.

What the AUC actually measures

The rank form is the most useful way to read the number: an AUC of 0.82 says that if you draw one positive and one negative sample at random, the model gives the positive the higher score 82% of the time (ties counted as half). That framing makes two properties obvious. First, the AUC only cares about ordering, not about the score values — multiply every score by 10, or push them all through a monotone squashing function, and the AUC does not move by a thousandth. Second, it is a whole-curve summary: two models can share an AUC of 0.80 while one is far better in the low-FPR corner where you actually plan to operate, so read the curve, not only the area.

The flip side is what the AUC hides. It says nothing about calibration — a model whose probabilities are all squeezed between 0.48 and 0.52 can still score 0.99 if the ordering is right, which is why probability outputs deserve a separate check (the logprob → probability calculator is handy when you are working from raw model logits). It also says nothing about the threshold you will ship. Once you have picked one, the per-threshold counts belong in a confusion matrix calculator so you can read precision, recall and specificity at that exact cut-off.

Edge cases this tool handles explicitly

• Only one class present. If every label is 1 (or every label is 0) then P·N = 0, the denominator of both formulas vanishes and the AUC is undefined. scikit-learn raises a ValueError here; this tool refuses the input with the same reasoning rather than printing a misleading 0.5.
• Every score tied. The sweep produces a single threshold, the curve becomes the diagonal, and both formulas return exactly 0.5 — the no-skill baseline. See the third worked example below for the arithmetic.
• AUC below 0.5.This is not a broken model so much as a reversed one. If the AUC comes out at 0.23, switching the “lower score = positive” direction gives 1 − 0.23 = 0.77. The usual causes are a positive-class label mix-up or a distance metric being fed where a similarity was expected.
• Tiny samples. With four or five rows the AUC can only take a handful of discrete values (with P = N = 2 it is one of 0, 0.25, 0.5, 0.75, 1) and its confidence interval is enormous. Treat single-digit-sample AUCs as illustrations, not evidence.
• Duplicate rows and near-ties. Floating-point scores that differ in the fifteenth decimal are not ties and will produce a visible extra ROC step. Round your scores before pasting if you intend them to tie.

The AUC is a model-selection number. Shipping a classifier needs one more decision: the cut-off at which a score becomes a positive prediction. The threshold table under the chart exists for exactly that step, and the row flagged as Youden-optimal is a default, not an answer.

Youden's J maximises TPR − FPR, which quietly assumes a false positive and a false negative cost the same and that the two classes are balanced. Both assumptions are usually wrong. When the costs differ, pick the threshold that minimises expected cost instead:

cost(t) = c_FN · P · (1 − TPR(t)) + c_FP · N · FPR(t)

Read TPR(t) and FPR(t)straight out of the threshold table, plug in your own two costs, and take the row with the smallest total. A loan-default model where a missed default costs fifty times a wasted review call will land on a much lower threshold than Youden's J suggests. A spam filter, where a false positive buries someone's payslip email, moves the other way.

Three constraints show up often enough to be worth naming. A capacity constraint fixes how many positives you can act on — if the review team handles 50 cases a day, sort by score and cut at the 50th, then read the resulting TPR off the table. A sensitivity floor is the medical-screening pattern: fix TPR at, say, 0.95 and accept whatever FPR that costs. A precision floor is the alerting pattern: nobody trusts a pager that cries wolf, so hold precision above a set level and take the recall you can get.

Once a threshold is chosen, stop reporting the AUC alone. The single most useful follow-up is the confusion matrix at that cut-off, from which the F1 score calculator gives the precision–recall balance, and the Matthews correlation coefficient calculator gives a single balanced score that, unlike accuracy, does not collapse when 96% of the data is one class. For ranking and retrieval work — search results, recommendations, RAG candidate lists — the ROC is the wrong instrument entirely, and precision@K and recall@K describe what users actually see.

Finally, treat one AUC number as an estimate with error bars. Hanley and McNeil give a standard error that widens sharply as the sample shrinks, so two models scoring 0.81 and 0.83 on a 200-row validation set are not meaningfully different. Cross-validate, report the spread across folds, and re-check the curve on data from the period you plan to deploy in — score distributions drift, and a threshold tuned on last quarter's data can silently move to a different point on the curve.

• scikit-learn — sklearn.metrics.roc_auc_score (reference TPR/FPR sweep and trapezoidal AUC)
• scikit-learn — sklearn.metrics.roc_curve (threshold sweep and ROC points)
• Fawcett, T. (2006) — An introduction to ROC analysis, Pattern Recognition Letters 27(8)
• Hanley & McNeil (1982) — The Meaning and Use of the Area under a ROC Curve, Radiology 143(1)

The formulas on this page were last cross-checked against these sources on 2026-06-10. ROC and AUC are stable mathematical definitions, so this tool needs no rate or schedule updates — only the worked examples are periodically re-reconciled against scikit-learn.