Brier Score Calculator

The Brier score grades probabilistic forecasts. Instead of asking whether a hard label was right, it measures how far each predicted probability sat from what actually happened. It was defined by Glenn Brier in 1950 for weather verification and is identical to the mean squared error of the probabilities — the same quantity scikit-learn returns from brier_score_loss.

With N forecasts, each a probability fᵢ ∈ [0, 1] of a binary event whose actual outcome is oᵢ ∈ {0, 1}:

BS = (1/N) Σ (fᵢ − oᵢ)²

1. Validate. Every probability must lie in [0, 1], every outcome must be 0 or 1, and the two lists must be the same length. Bad input gets a specific message, never a silent NaN.
2. Score. Square each gap (fᵢ − oᵢ)² and average them. For binary outcomes this lands in [0, 1]; 0 is a perfect, fully-confident-and-correct forecaster.
3. Baseline. Compute the reference Brier score for a constant forecast r:

   BS_ref = (1/N) Σ (r − oᵢ)²

   With the base rate r = ō (the mean outcome) this simplifies to the outcome variance ō(1 − ō) — the score of a climatology forecaster that always predicts the long-run frequency.
4. Skill. The Brier Skill Score rescales the Brier score against that baseline:

   BSS = 1 − BS / BS_ref

   Above 0 the model beats the baseline; 0 ties it; below 0 it is worse than just predicting the reference. When the baseline is itself perfect (BS_ref = 0, every outcome identical) the skill score is undefined and the tool shows “—” rather than dividing by zero.

As an internal correctness gate the tool also recomputes the Brier score a second way — splitting the sum by class into Σ(1 − fᵢ)² over the positive cases plus Σfᵢ² over the negatives — and asserts the two agree to floating-point precision. The two forms are algebraically identical because oᵢ²= oᵢ for binary outcomes, so any disagreement would signal a bug.

What the score is actually made of

Allan Murphy showed in 1973 that the Brier score splits into three interpretable pieces once you group forecasts by the distinct probability values they used. With K distinct forecast values, n_k cases issued at value f_k, and ō_k the observed frequency inside that group:

BS = reliability − resolution + uncertainty

• Reliability = (1/N) Σ n_k (f_k − ō_k)² — miscalibration. When you say 70% and the event happens 70% of the time, this term is 0. Lower is better.
• Resolution = (1/N) Σ n_k (ō_k − ō)² — how far your groups pull away from the overall base rate. This is sharpness that turned out to be justified. Higher is better, and it subtracts from the score.
• Uncertainty = ō(1 − ō) — the difficulty of the events themselves. You cannot change it by forecasting better; it is the same term that acts as the base-rate baseline.

The decomposition explains why two very different failures produce the same headline number. A model that is perfectly calibrated but never moves off the base rate has zero reliability error and zero resolution — it scores exactly the baseline. A model that is sharp but overconfident has strong resolution eaten back by a large reliability term. If you want the reliability half measured directly rather than inferred, the expected calibration error calculator bins the same probabilities and reports the calibration gap on its own.

Edge cases this tool handles

• All outcomes identical. If every oᵢ is 1 (or every one is 0) the base rate is 1 (or 0), so BS_ref = ō(1 − ō) = 0 and the skill score would divide by zero. The tool returns “—” with a note instead of NaN or Infinity.
• Probabilities pasted as percentages. A value like 90 is rejected by name and position rather than silently scored — the Brier score of a 90 against an outcome of 1 would be 7921, which is meaningless. Divide by 100 first.
• Mismatched list lengths. Twelve probabilities and eleven outcomes is the most common paste error in evaluation work. The error message names both counts so you can see which column lost a row.
• Outcomes that are not 0 or 1.This is a binary scorer, so a stray 2, “yes” or empty cell is refused. Map labels to 0/1 before pasting.
• Single forecast. One pair is legal — the score is just that one squared error — but the base-rate skill score is undefined for it, because a one-case base rate is always 0 or 1.
• Brackets and newlines. Square brackets, parentheses, tabs, commas and line breaks are all stripped, so a NumPy array or a spreadsheet column pastes in directly with no cleanup.

Two practical cautions. First, the Brier score is only comparable across datasets with the same base rate — a 0.05 on a rare-event problem may be worse than a 0.20 on a balanced one, which is exactly why the skill score exists. Second, with small N the score is noisy: twenty forecasts give you an estimate with a wide interval around it, so treat rankings between close models as provisional until you have a few hundred cases.

The Brier score answers one question: were the probabilities themselves right? Most of the metrics sitting next to it in an evaluation report answer something else, and reading them together is how you tell a ranking problem from a calibration problem.

• Log loss / cross-entropy — the other strictly proper rule. It punishes confident errors without bound where the Brier score caps a single case at 1. Score the same file both ways with the cross-entropy loss calculator; when the two disagree about which model wins, a few extreme predictions are driving the result.
• ROC AUC — pure ranking. AUC is unchanged if you put every probability through any monotone transform, so a badly calibrated model can post a 0.95 AUC and a poor Brier score at the same time. Pair this page with the ROC AUC calculator to separate discrimination from calibration.
• Accuracy, precision, recall and F1 — threshold metrics. They need a cut-off before they mean anything, and they discard confidence entirely. If you have thresholded predictions rather than probabilities, the F1 score calculator and the confusion matrix calculator are the right pages for that stage.
• Expected calibration error — the reliability term of the Brier decomposition, measured directly and reported on its own scale. Useful when you need to say how far off the probabilities are rather than how much total error they caused.

A reasonable default in a model card: report the Brier score, the Brier Skill Score against the base rate, and one ranking metric. The first says how good the probabilities are, the second says whether they beat doing nothing, and the third says whether the ordering is usable even if the numbers need recalibrating.

• Brier, G. W. (1950) — Verification of Forecasts Expressed in Terms of Probability, Monthly Weather Review 78(1)
• scikit-learn — sklearn.metrics.brier_score_loss (the binary implementation matched here)
• NOAA / US National Weather Service — Forecast Verification Glossary (Brier Skill Score)
• Murphy, A. H. (1973) — A New Vector Partition of the Probability Score, Journal of Applied Meteorology 12(4) (the reliability / resolution / uncertainty decomposition)

The formulas on this page were last cross-checked against these sources on 2026-09-07. The Brier score is a stable mathematical definition, so this tool needs no rate or schedule updates — only the worked examples are periodically re-reconciled against scikit-learn.