AI Fill-Mask Word Predictor

This tool is a live demonstration of masked language modelling (MLM), the pre-training objective introduced for BERT by Devlin, Chang, Lee & Toutanova (2018). It is the exact task BERT was trained on — recover a hidden word from its context — exposed as something you can type into.

1. Tokenise and locate the blank

Your sentence is split into WordPiece tokens by the model's tokenizer, which wraps it with a [CLS] start token and a [SEP] end token. The position of your [MASK] token is recorded — that is the only position the model predicts.

2. One forward pass, one logit per word

The tokens pass through BERT's bidirectional Transformer encoder, producing a context vector at the masked position that has absorbed information from every other word — left and right. That vector goes through the MLM head, which emits a logit (a raw, unbounded score) for every one of the roughly 30,522 words in the vocabulary.

3. Softmax, then top-k

The logit vector is turned into a probability distribution with the softmax function pᵢ = exp(zᵢ) / Σⱼ exp(zⱼ), so every word gets a probability and they sum to 1 across the whole vocabulary. The tool sorts descending and shows the top-k. There is no temperature and no sampling, so the same sentence, model, and k always give the same ranking.

tokens   = tokenizer("[CLS] The capital of France is [MASK] . [SEP]")
h_mask   = BERT(tokens)[mask_position]        # bidirectional context
logits   = MLM_head(h_mask)                    # one score per vocab word
probs    = softmax(logits)                      # sum to 1 over ~30,522 words
top_k    = argsort(probs, desc)[:k]             # greedy, reproducible

4. Server-side inference & the shown shares

The model runs on the Hugging Face Inference API — never in your browser — so no hundreds-of-megabytes weight file is downloaded. The shown words rarely sum to 100% on their own because most of the probability mass is spread thinly across thousands of unlikely words; the tool therefore reports how much of the total mass your top-k capture and renormalises the shown words into shares that do sum to 100% for the confidence bars. If the Inference API is out of quota or unreachable, a deterministic grammar baseline — a small local-context frequency table — fills in so the page never dead-ends, clearly labelled so you know it is a fallback and not the neural model.

• Devlin, Chang, Lee & Toutanova (2018) — BERT: Pre-training of Deep Bidirectional Transformers (arXiv)
• Hugging Face — Fill-Mask task page (pipeline, softmax + top-k spec)
• Hugging Face — google-bert/bert-base-uncased (model card, WordPiece vocabulary)
• Hugging Face — distilbert/distilbert-base-uncased (distilled model card)
• Hugging Face — Inference Providers documentation (server-side runtime)

The MLM method, model cards, and Inference API endpoint were last cross-checked on 2026-07-11. The page is reviewed whenever an upstream model repository changes or a new recommended fill-mask checkpoint is published.