AI Fill-Mask Word Predictor

This tool is a live demonstration of masked language modelling (MLM), the pre-training objective introduced for BERT by Devlin, Chang, Lee & Toutanova (2018). It is the exact task BERT was trained on, recover a hidden word from its context, exposed as something you can type into.

1. Tokenise and locate the blank

Your sentence is split into WordPiece tokens by the model's tokenizer, which wraps it with a [CLS] start token and a [SEP] end token. The position of your [MASK] token is recorded, that is the only position the model predicts.

2. One forward pass, one logit per word

The tokens pass through BERT's bidirectional Transformer encoder, producing a context vector at the masked position that has absorbed information from every other word, left and right. That vector goes through the MLM head, which emits a logit (a raw, unbounded score) for every one of the roughly 30,522 words in the vocabulary.

3. Softmax, then top-k

The logit vector is turned into a probability distribution with the softmax function pᵢ = exp(zᵢ) / Σⱼ exp(zⱼ), so every word gets a probability and they sum to 1 across the whole vocabulary. The tool sorts descending and shows the top-k. There is no temperature and no sampling, so the same sentence, model, and k always give the same ranking.

tokens   = tokenizer("[CLS] The capital of France is [MASK] . [SEP]")
h_mask   = BERT(tokens)[mask_position]        # bidirectional context
logits   = MLM_head(h_mask)                    # one score per vocab word
probs    = softmax(logits)                      # sum to 1 over ~30,522 words
top_k    = argsort(probs, desc)[:k]             # greedy, reproducible

4. Server-side inference & the shown shares

The model runs on the Hugging Face Inference API, never in your browser, so no hundreds-of-megabytes weight file is downloaded. The shown words rarely sum to 100% on their own because most of the probability mass is spread thinly across thousands of unlikely words; the tool therefore reports how much of the total mass your top-k capture and renormalises the shown words into shares that do sum to 100% for the confidence bars. If the Inference API is out of quota or unreachable, a deterministic grammar baseline, a small local-context frequency table, fills in so the page never dead-ends, clearly labelled so you know it is a fallback and not the neural model.

• Devlin, Chang, Lee & Toutanova (2018): BERT: Pre-training of Deep Bidirectional Transformers (arXiv)
• Hugging Face: Fill-Mask task page (pipeline, softmax + top-k spec)
• Hugging Face: google-bert/bert-base-uncased (model card, WordPiece vocabulary)
• Hugging Face: distilbert/distilbert-base-uncased (distilled model card)
• Hugging Face: Inference Providers documentation (server-side runtime)

The MLM method, model cards, and Inference API endpoint were last cross-checked on 2026-07-11. The page is reviewed whenever an upstream model repository changes or a new recommended fill-mask checkpoint is published.