METEOR Score Calculator

METEOR (Metric for Evaluation of Translation with Explicit ORdering), defined by Banerjee & Lavie (2005), scores a candidate against a reference by first aligning their words and then balancing how much content is shared against how well the word order is preserved. Unlike BLEU, it rewards recall heavily and adds an explicit penalty for jumbled output. The score is:

METEOR = Fmean · (1 − Pen)

It is built in four steps from the tokenised, optionally lowercased texts:

1. Align unigrams. Build the largest one-to-one mapping between candidate and reference words. This tool uses exact matching and, optionally, a Porter-stem stage applied to the words left over after the exact pass, the same exact/stem ordering as NLTK's meteor_score. WordNet synonym matching is out of scope here.
2. Precision and recall. With m mapped unigrams, candidate length c and reference length r, P = m/c and R = m/r.
3. Fmean. A recall-weighted harmonic mean, Fmean = (P·R)/(α·P + (1 − α)·R). With the default α = 0.9 this equals 10·P·R/(R + 9·P), so recall pulls nine times harder than precision.
4. Fragmentation penalty. Group the mapped unigrams into the fewest chunks, runs adjacent in both the candidate and the reference. With ch chunks over m matches, Pen = γ·(ch/m)^β using γ = 0.5 and β = 3. Many short chunks (scrambled word order) drive the penalty up; one long chunk barely dents the score.

The α = 0.9, γ = 0.5 and β = 3 defaults are the values from Banerjee & Lavie (2005), confirmed against NLTK's single_meteor_score defaults; Lavie & Agarwal (2007) discuss tuning them per language. Fmean is computed as the direct ratio and independently re-derived as the reciprocal 1/(α/R + (1 − α)/P); when the two agree the score is flagged “cross-checked”. One quirk worth knowing: two identical sentences score about 0.998, not 1, because a single chunk still incurs Pen = γ·(1/m)^β, a correct, documented property of METEOR.

Tokenisation and casing, the usual reason two tools disagree

Nothing is matched until both texts are tokenised. Here, runs of letters and digits become word tokens and every other character becomes a token of its own, so the mat. is three tokens, not two. Lowercasing is on by default. Both choices move the score, because METEOR compares surface forms: with lowercasing off, “The Cat Sat” against “the cat sat” finds no matches at all and scores 0.0000, and one trailing full stop on an otherwise identical six-word pair takes it from 0.9977 down to 0.9813. When your number will not reconcile with another implementation, check the tokeniser before you question the formula. The same holds for the BLEU calculator and every other overlap metric.

Edge cases, and what this tool does with them

• No matches at all. With m = 0 both P and R are 0 and Fmean divides by zero. METEOR is defined as 0 in that case, which is what NLTK returns and what you get here, with the alignment view showing every candidate token unmatched.
• Empty candidate or reference. An empty side is not a score of 0, it is no score at all. The tool stops and tells you which box is empty rather than publishing a misleading zero.
• Repeated words. The alignment is greedy, not exhaustive, so a repeated reference word can be mapped to the far copy. Scoring “the cat sat” against “the cat sat on the mat” matches all three candidate words, but the candidate's “the” takes the reference's second “the” at index 4, which splits the match into two chunks: Pen = 0.5·(2/3)³ = 0.1481 and METEOR = 0.4483, where a single chunk would have given 0.5166. NLTK does exactly the same thing. It is the price of a greedy alignment, and worth knowing before you file a bug against your own evaluation script.
• Length mismatch. METEOR has no brevity penalty of BLEU's kind. The recall term does that job, and because α = 0.9 it does it asymmetrically: dropping content costs far more than adding it. The third and fourth worked examples below put numbers on that.
• Custom parameters. α and γ are clamped to [0, 1] and β to non-negative values, and anything unparseable falls back to the published default, so a typo in the Advanced panel cannot produce NaN or a score outside 0 to 1.

From one sentence to a whole test set

This calculator scores one sentence pair, which is the unit METEOR was designed to correlate with human judgment on. Two different conventions exist for reporting a system-level number, and they do not agree. The 2005 paper aggregates the sufficient statistics over the whole test set first, summing m, c, r and ch across every sentence and applying the formulas once; most modern toolkits instead report the arithmetic mean of the per-sentence scores. Aggregated statistics let long sentences pull harder, a sentence mean treats every sentence alike. Neither is wrong, but say which one you used, and never compare a mean against an aggregate. For a quick character-level sanity check on the same data, the chrF calculator is less sensitive to the tokeniser than any word-level metric.

No single overlap metric is trustworthy on its own. The usual practice is to report two or three together and look at where they disagree, because each one is blind to something different:

• BLEU counts matching n-grams up to length four and applies a brevity penalty. It is precision-led, has no notion of stems or synonyms, and is noisy on a single sentence, which is why it is quoted at corpus level. Where METEOR forgives padding and punishes dropped content, BLEU leans the other way.
• ROUGE was built for summarisation and is recall-oriented. ROUGE-L uses the longest common subsequence rather than chunks, so it rewards word order more gently than METEOR's fragmentation penalty does. It shares the Porter stemmer used in this tool's stem mode.
• chrF scores character n-grams instead of words, which makes it the safer choice for morphologically rich languages, Sinhala and Tamil included, where one wrong suffix destroys a word-level match that was almost right.
• WER is an edit-distance metric, (substitutions + deletions + insertions) divided by reference length, where lower is better and the scale runs past 1. It is the standard for speech recognition and ignores meaning entirely.

Pick METEOR when you want sentence-level agreement with human raters and credit for inflected forms. Pick BLEU when you need a number other papers will recognise without explanation. Report both when you are claiming an improvement, and publish your tokenisation alongside the scores, because a METEOR figure without its tokeniser and matching mode cannot be reproduced by anyone, including you in six months.

METEOR measures overlap with one reference, not translation quality. A candidate can be a perfectly good translation and score poorly simply because the reference phrased it differently, and a candidate can score well while reversing the meaning of the sentence, since negation is a single short word. Three practical limits to keep in mind:

• One reference sets the ceiling. The more valid ways there are to say something, the more a single reference understates every system. If scores look oddly low across the board, the reference set is usually the suspect.
• Short sentences are unstable. The penalty term depends on ch/m, so with m = 2 or m = 3 a single chunk break moves the score by a tenth or more. Judge systems on a few hundred sentences, not on one.
• Synonyms are out of scope here. This version runs the exact and Porter-stem stages only. A paraphrase that uses different words for the same idea will be marked down, so for semantic closeness rather than surface overlap, an embedding-based check is the better instrument.

Everything is computed in your browser. No candidate text, reference text or score leaves the page, which matters when the sentences you are evaluating come from a client's data.

• Banerjee & Lavie (2005): METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments (ACL Workshop)
• NLTK nltk.translate.meteor_score: reference implementation (single_meteor_score, default α/γ/β)
• Lavie & Agarwal (2007): METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments (WMT)
• Porter, M. F. (1980): An algorithm for suffix stripping (the Porter stemmer used in stem mode)

The formulas and the worked examples on this page were last reconciled against Banerjee & Lavie (2005) and NLTK single_meteor_score on 2026-06-12. The calculation module ships with a built-in assertion that re-runs every worked example, so a regression in the METEOR math fails fast.