TF-IDF Calculator

TF-IDF(term frequency–inverse document frequency) scores how important a word is to one document within a collection. A word that appears often in a document but rarely across the corpus gets a high score; a word that appears everywhere gets a low one. The definitions here follow Manning, Raghavan & Schütze's Introduction to Information Retrieval, Chapter 6, and scikit-learn's TfidfVectorizer.

The tool computes it in four steps:

1. Tokenise. Each line is split on whitespace into unigrams. With the default toggle on, tokens are lower-cased and stripped of leading and trailing punctuation, then a sorted vocabulary is built from every document.
2. Term frequency. raw uses the count itself; relative divides by the document length; and sublinear uses 1 + ln(count), damping very frequent words — the same option as scikit-learn's sublinear_tf.
3. Inverse document frequency. The document frequency df is how many documents contain the term. Standard idf is log_b(N / df) for base e, 10, or 2. The scikit-learn smoothed form is ln[(1 + N) / (1 + df)] + 1; the +1 inside avoids dividing by zero, and the trailing +1 stops a term that appears in every document from being zeroed out.
4. Multiply and optionally normalise. Each weight is tf × idf. Turning on L2-normalisation divides each document's column by its Euclidean norm, so every document vector has unit length — required to match TfidfVectorizer's default norm='l2'.

One subtlety worth knowing: scikit-learn additionally discards single-character tokens and uses a regex tokeniser, so for very short words its vocabulary can differ slightly from this tool's plain whitespace split. For the toy corpora students usually check, the two agree once you select raw TF, smoothed IDF, and L2-normalisation. As a credibility check, the calculator re-derives every idf a second, independent way — the subtraction form (log N − log df) ÷ log b — and confirms the two routes agree. The optional cosine-similarity matrix then reuses the same vectors to show how alike the documents are.

The formula in one line

For a term t in document d inside a corpus of N documents: tfidf(t, d) = tf(t, d) × log(N / df(t)). Everything else — smoothing, log base, sublinear damping, L2 normalisation — is a variation on which tf you count and which log you take. That is why two textbooks, two lecturers and scikit-learn can all print different numbers for the same three sentences and all of them be right. This calculator shows you the intermediate df and idf columns precisely so you can tell which convention produced the answer in front of you.

Does the log base matter?

Only for scale. Changing from base e to base 2 multiplies every idf by 1⁄ln 2 ≈ 1.4427, and changing to base 10 divides every idf by ln 10 ≈ 2.3026. Because the same constant hits every term, the ranking of terms within a document never changes, and after L2 normalisation the vectors are identical whichever base you picked. Information-theory courses tend to prefer base 2, so a weight reads as “bits of surprise”; scikit-learn, NLTK and gensim all use the natural log. Pick the base your lecturer used and stay with it — mixing bases mid-assignment is the single most common reason a hand calculation refuses to match the library.

Tokenisation is half the answer

Before any arithmetic happens, the text has to become tokens, and the split you choose changes dffor every term. This tool splits on whitespace and, with the lower-case toggle on, strips leading and trailing punctuation — so Cat, cat and cat, collapse to one token. scikit-learn instead applies the regex (?u)\b\w\w+\b, which silently drops every one-character token, so a corpus full of “a” and “I” will produce a smaller vocabulary there than here. If you want to see how a modern language model would cut the same sentence into sub-word pieces instead of whole words, the AI tokenizer visualizer shows the split token by token, and the n-gram generator builds the bigrams and trigrams you would feed in if you wanted phrase features rather than single words.

Edge cases this calculator handles explicitly

• A term in every document. df = N, so standard idf = log(1) = 0 and the weight is 0 everywhere. Smoothed idf returns 1 instead, leaving the weight equal to the tf.
• A single-document corpus. Every term has df = N = 1, so the entire standard matrix is zeros. The engine flags this case so the interface can say why, rather than showing a silent wall of 0.0000.
• A term absent from a document.count = 0 gives tf = 0 under all three schemes — including sublinear, where the 1 + ln(count) rule is only applied when the count is at least 1, since ln(0) is undefined.
• Sublinear tf of a single occurrence. 1 + ln(1) = 1, so a word used once scores the same as under raw counts; the damping only bites from the second occurrence onwards.
• Text that tokenises to nothing. A line of pure punctuation produces an empty vocabulary, and the tool says so instead of dividing by zero.
• Oversized input. The corpus is capped at 20 documents, 2,000 characters per line and 20,000 overall, because this is a teaching and checking tool rather than a production indexer.

Once the matrix exists, each document is a vector in vocabulary space, and the natural next question is how close two of those vectors are. The optional similarity matrix uses the cosine of the angle between them, which is the standard measure for TF-IDF vectors because it ignores document length; the cosine similarity calculator shows that same computation on its own if you want to feed in vectors you produced elsewhere.

TF-IDF is the first ranking signal most information-retrieval courses teach, and it is still a live baseline rather than a museum piece. It helps to know what sits either side of it.

Plain word frequencyis simpler and sometimes enough. If all you need is which words dominate one page — for an SEO audit, say, rather than a corpus comparison — a keyword density checker answers that directly, with no second document required. TF-IDF only becomes meaningful once you have a corpus to contrast against, because the entire idf half of the formula is a statement about the other documents.

BM25is the tuned successor. It keeps the same idf idea but replaces the linear tf with a saturating one controlled by a parameter k₁, so the fifth occurrence of a word adds far less than the first, and it corrects for document length with a second parameter b. Lucene, Elasticsearch and OpenSearch all rank with BM25 by default, so if your goal is to reproduce a real search engine’s scores rather than a textbook’s, the BM25 score calculator is the closer match.

Dense embeddingssit at the other end. They map a whole sentence into a few hundred learned dimensions, so “car” and “automobile” land near each other where TF-IDF treats them as unrelated columns. The trade is explainability and cost: an embedding weight cannot be traced back to a count, and you need a model to produce one. Most production search stacks now run both and blend the scores, which is exactly why the sparse, countable half is still worth understanding on its own terms.

Coursework and exam prep. The most common visitor here is a student with a three-sentence corpus, a worked answer from a lecture slide, and a mismatch they cannot explain. Nine times out of ten the cause is a convention, not an arithmetic slip: relative versus raw tf, natural versus base-2 log, smoothed versus standard idf, or normalisation applied at the end. Because this page prints the df and idf columns separately, you can find the exact step where your working diverged instead of re-deriving the whole table.

Keyword extraction. Rank the terms of one document by TF-IDF against a background corpus and the top handful usually reads like a sensible tag list. It is a cheap, deterministic first pass before reaching for a model, and it is why the technique still turns up inside content tooling, tagging pipelines and document classifiers.

Retrieval and RAG.Sparse TF-IDF or BM25 retrieval catches exact identifiers — part numbers, NIC formats, error codes, proper nouns — that dense vectors routinely miss. Hybrid retrievers run a sparse and a dense index side by side and merge the rankings, which keeps the literal-match strength of counting while gaining the semantic reach of embeddings.

Deduplication and clustering. Turn each document into a TF-IDF vector, take the cosine between every pair, and near-duplicates surface as values close to 1. That is the whole basis of a lot of plagiarism screening and near-duplicate detection, and it is why this tool ships the similarity matrix alongside the weights.

• Manning, Raghavan & Schütze — Introduction to Information Retrieval, Ch. 6: tf, df, idf = log(N/df), and tf-idf weighting
• scikit-learn — Tf–idf term weighting: the smoothed idf formula, sublinear_tf, and L2 normalisation

The formulas on this page were last cross-checked against these sources on 2026-08-28. TF-IDF is a stable mathematical definition, so this tool needs no rate or schedule updates — only the worked examples are periodically re-reconciled against scikit-learn.