N-gram Generator — Word & Character n-grams

An n-gram is a contiguous run of nitems pulled from a sequence. This tool builds n-grams two ways — over words or over characters — using the classical sliding-window construction described in Jurafsky & Martin's Speech and Language Processing, §3 "N-gram Language Models". Every step is a deterministic string operation, so the output is fully reproducible and never leaves your device.

1. Tokenise. In word mode the text is optionally lowercased, optionally stripped of punctuation (intra-word apostrophes and hyphens are kept, so don't and well-being survive), then split on whitespace. In character mode each word's letters become the sequence, and n-grams are formed within a word so they never straddle the gap between two words — matching scikit-learn's char_wb intent.
2. Slide the window. For a sequence of length T and window size n, the tool emits the items at positions i … i+n−1 for every start i from 0 to T−n. The number of n-grams in a segment is therefore max(0, T − n + 1).
3. Count. Identical n-grams are tallied into a frequency map. Word n-grams are shown joined by a single space; character n-grams are concatenated. The table is sorted by count descending, with ties broken alphabetically so the same text always produces the same order.
4. Optional sentence padding. When enabled (word mode only), the text is first split into sentences on . ! ? boundaries, and each sentence is padded with n−1 leading <s> markers and one trailing </s> marker. This mirrors NLTK's pad_left/pad_right preprocessing and is what language-model training uses to learn sentence starts and ends.

Every count is independently cross-checked: after the sliding window builds the list, the total is re-derived straight from the closed-form Σ max(0, T − n + 1) over each segment. When the two agree — they always should — the badge in the tool reads "count verified". What this tool deliberately does not do is estimate probabilities or apply smoothing (Laplace, Kneser-Ney): it is an n-gram extractor, not a language model, so the numbers you see are exact observed counts, nothing modelled.

• Jurafsky & Martin — Speech and Language Processing (3rd ed. draft), Ch. 3: N-gram Language Models
• NLTK — nltk.util.ngrams and sentence padding (pad_left / pad_right)
• scikit-learn — CountVectorizer (char / char_wb analyzers, ngram_range)

The construction and the worked examples on this page were last reconciled against these sources on 2026-07-13. The definitions are standard and stable; the page is reviewed if the reference implementations change their default behaviour.