What is the NIST score in machine translation?

NIST is an automatic metric for machine-translation quality (Doddington, 2002). It compares a candidate translation with one or more references and rewards matching n-grams, weighting each match by how informative (rare) it is. The information-weighted precisions for orders 1 to N are summed and multiplied by a brevity penalty, giving an unbounded, non-negative score.

How is NIST different from BLEU?

Two ways. First, BLEU treats every matched n-gram equally, while NIST weights each match by its information content — log2 of how much the (n-1)-gram prefix narrows down the next word — so rare, meaningful matches count for more than common ones. Second, BLEU takes the geometric mean of its precisions (one zero order forces the whole score to zero); NIST sums its precisions, so a missing higher order just adds nothing.

What is a good NIST score?

There is no fixed threshold. Because NIST is unbounded and depends on N, the tokenisation, and the reference set, a value is only meaningful relative to other systems scored the same way. Strong sentence-level systems often fall in roughly the 5–12 range, but you should compare NIST scores within one experiment, not against an absolute cutoff.

How is the NIST information weight calculated?

For an n-gram w1…wn, Info = log2( count(w1…w(n-1)) / count(w1…wn) ), where the counts come from the reference set. For a unigram the prefix is empty, so the numerator is the total number of reference n-grams. A word that always follows its prefix carries weight 0; a rare word carries a high weight. Counts guarantee the ratio is at least 1, so weights are never negative.

Does NIST use a brevity penalty like BLEU?

Yes, but a different one. NIST's penalty is BP = exp( beta · (ln ratio)² ) for a length ratio below 1, with beta fixed so BP = 0.5 at a two-thirds ratio (beta is about -4.216). A candidate at least as long as the reference is not penalised. BLEU instead uses exp(1 - r/c). NIST's penalty falls off more gently near a ratio of 1 and more sharply for very short output.

Does this match NLTK's sentence_nist?

Yes. The information weights, the summed per-order precisions, the best-reference selection, and the mteval-v13a length penalty all follow nltk.translate.nist_score. One deliberate difference: NLTK raises a ZeroDivisionError when the candidate is shorter than N; this tool treats those empty orders as contributing 0 and shows a note, so you always get a number.

Which tokeniser should I use?

The NIST-regex option reproduces the mteval-v13a tokeniser — it splits punctuation, keeps decimals like 3.14 and thousands like 1,000 intact, and lowercases when that toggle is on. Use it to match the official scorer. The whitespace option splits only on spaces and is handy when your text is already tokenised. Keep the same choice across every system you compare.

Can I score against more than one reference?

Yes. Put one reference per line. For each n-gram order the tool uses the single reference that gives the largest information-weighted match, and the brevity penalty uses the accumulated reference length, exactly as NLTK does. More references usually raise the score because there are more legitimate ways for the candidate to match.

Why did my NIST score not drop to zero when BLEU did?

NIST sums its per-order precisions instead of taking a geometric mean. If your candidate has no 4-gram or 5-gram matches, those orders contribute 0 to the NIST sum but the unigram and bigram scores still stand. Under BLEU-4 the same missing order would zero the entire score. This is why short candidates often get a usable NIST number but a BLEU of 0.

AI · Machine Translation

NIST Score Calculator

Paste a candidate translation and one or more references to get the NIST score, the information-weighted precision for each n-gram order 1–5, the NIST brevity penalty, and a match inspector that shows every n-gram's information weight — so you can see exactly how the number was derived. Matches NLTK sentence_nist, no signup, runs in your browser.

By Induwara Ashinsana— Executive Director, Ryzera TechnologiesUpdated Jul 13, 2026

NIST Score

Candidate (machine output)

One sentence. Stays on your device.0 words

Reference(s) — one per line

At least one. Stays on your device.0 words

Max n-gram (N)

Examples

NIST-5

—

Precision sum

—

Brevity penalty

—

Candidate length

—

Runs entirely in your browser — your text is never uploaded, logged, or stored. Method: information-weighted n-gram precision summed over orders 1–N with the NIST brevity penalty, per Doddington (2002); reconciled to NLTK sentence_nist. Up to 50,000 characters per box.

How it works

NIST is the information-weighted cousin of BLEU, introduced by George Doddington (2002) for the NIST machine-translation evaluations. Like BLEU it compares a system's output (the candidate) against one or more correct referencetexts, but it makes two changes that address BLEU's known weaknesses. The final score is:

NIST = ( Σ scoreₙ, n = 1…N ) · BP

Information weights. Every reference n-gram gets a weight measuring how much information it carries: Info(w₁…wₙ) = log₂( count(w₁…w_{n-1}) ÷ count(w₁…wₙ) ). The counts are taken over the reference set. For a unigram the prefix is empty, so the numerator is the total number of reference n-grams. A predictable word (one that almost always follows its prefix) scores near 0; a rare, surprising word scores high. This is what makes NIST reward content words over function words.
Information-weighted precision per order.For each order n from 1 to N, the tool sums the information weight of every candidate n-gram that also appears in a reference (clipped to the reference's count), then divides by the number of candidate n-grams of that order: scoreₙ = Σ Info(matched) ÷ (candidate n-grams). With several references, the reference giving the largest weighted match is used for that order.
Sum, don't average. BLEU takes the geometric mean of its precisions, so a single order with no match forces the whole score to 0. NIST simply adds the per-order scores, so a missing 4-gram only means that order contributes nothing — the lower orders still count. This is why NIST is more stable on short sentences.
Brevity penalty. With Lsys the candidate length and Lref the (accumulated) reference length, and ratio = Lsys / Lref: BP = exp( β · (ln ratio)² ) when the candidate is shorter than the reference, and BP = 1 when it is at least as long. β = ln(0.5) ÷ (ln 1.5)² ≈ −4.216 is fixed so the penalty is exactly 0.5 at a two-thirds length ratio, matching mteval-v13a.

The brevity penalty is computed two independent ways — the mteval piecewise function and the closed-form β formula — and flagged “cross-checked” when they agree. The numbers reconcile with NLTK's sentence_nist when the tokenisation matches. Because NIST is unbounded and shifts with N and the reference count, always report it together with those settings.

Worked examples

One substitution, NIST-1 → 1.8208

Candidate: the cat sat on the rug
Reference: the cat sat on the mat

Reference 'the cat sat on the mat': 6 tokens, 'the' ×2
Info(the) = log2(6/2) = 1.5850; Info(cat/sat/on/mat) = log2(6/1) = 2.5850
Matched: the ×2, cat, sat, on ('rug' has no match)
info sum = 2·1.5850 + 3·2.5850 = 10.9248; ÷ 6 candidate words = 1.8208
Lsys = 6, Lref = 6 → ratio 1 → BP = 1
NIST-1 = 1.8208 × 1 = 1.8208

Too short, NIST-1 → 0.2970 (BP bites)

Candidate: the cat sat
Reference: the cat sat on the mat

Matched: the, cat, sat → info sum = 1.5850 + 2.5850 + 2.5850 = 6.7549
÷ 3 candidate words → score1 = 2.2516
Lsys = 3, Lref = 6 → ratio 0.5
BP = exp(−4.216 · (ln 0.5)²) = exp(−2.0257) = 0.1319
NIST-1 = 2.2516 × 0.1319 = 0.2970
Lower than the substitution case despite three exact matches — the brevity penalty is the point

Perfect match, NIST-5 → 4.3886

Candidate: the cat sat on the mat
Reference: the cat sat on the mat

N = 5, so totalRefWords = 6+5+4+3+2 = 20
score1 = (2·log2(20/2) + 4·log2(20/1)) / 6 = 23.9316 / 6 = 3.9886
score2 = (Info('the cat') + Info('the mat')) / 5 = (1 + 1) / 5 = 0.4000
score3 = score4 = score5 = 0 (every higher n-gram is fully predictable → weight 0)
BP = 1 (identical lengths)
NIST-5 = 3.9886 + 0.4000 = 4.3886 — note it exceeds 1, as NIST is uncapped

Frequently asked questions

Sources & references

The formulas and the worked examples on this page were last reconciled against Doddington (2002) and NLTK sentence_nist on 2026-07-13. The calculation module ships with a built-in assertion that re-runs every worked example, so a regression in the NIST math fails fast.

Related tools

LiveAI

METEOR Score Calculator

Calculate the METEOR score for a candidate translation against a reference, entirely in your browser. Shows unigram matches, precision, recall, the recall-weighted Fmean, the chunk-based fragmentation penalty, and the aligned tokens. Matches NLTK single_meteor_score, no signup.

Open tool

LiveAI

BLEU Score Calculator

Calculate the BLEU score for a candidate translation against one or more references, entirely in your browser. Shows the modified n-gram precisions p1–p4, the brevity penalty, and the clipped match counts. Matches NLTK sentence_bleu, no signup.

Open tool

LiveAI

TER Score Calculator

Calculate the Translation Edit Rate (TER) between a machine hypothesis and one or more references, entirely in your browser. Shows insertions, deletions, substitutions, block shifts, the edit breakdown, and a word-level alignment. Matches sacrebleu, no signup.

Open tool

Rate this tool

Be the first to rate

Comments & feedback

Spotted a bug or want an improvement? Tell us — our team reviews every comment, and good ideas get built. Comments are public and anonymous.

Found a bug, edge case, or want to suggest an improvement?

Email me at [email protected] — most fixes ship within 24 hours.