How do you calculate an attention score?

An attention score between a query and a key is their dot product: multiply the query and key vectors element by element and sum. In scaled dot-product attention you then divide that score by √dₖ (the square root of the head dimension) and pass the whole row through softmax to get weights that sum to 1.

Why is attention divided by the square root of dₖ?

For random query and key components with unit variance, a dot product over dₖ dimensions has variance dₖ, so its magnitude grows with dimension. Large scores push softmax into regions with tiny gradients. Dividing by √dₖ rescales the scores back to unit variance, keeping softmax gradients stable during training. This is from Vaswani et al., §3.2.1.

What is the difference between attention scores and attention weights?

Scores are the raw (or √dₖ-scaled) dot products QKᵀ — any real number, positive or negative. Weights are what you get after applying softmax to each row of scores: they are all between 0 and 1 and each row sums to exactly 1, so they act as a probability-weighted average over the value vectors.

How does softmax turn attention scores into weights?

For each query row, softmax exponentiates every score and divides by the row's total: weightⱼ = e^{scoreⱼ} / Σ e^{scoreₖ}. This tool subtracts the row maximum before exponentiating — a standard trick that avoids floating-point overflow without changing the result. The weights always sum to 1.

What are Q, K, and V in self-attention?

Q (query), K (key), and V (value) are three matrices, one row per token. The query asks 'what am I looking for?', each key advertises 'what do I offer?', and the value carries the content that gets mixed. In self-attention all three come from the same input sequence. This calculator takes Q, K, and V directly so you can see the arithmetic without learned projections.

What does the causal (decoder) mask do?

The causal mask sets every score where the key position is later than the query position (j > i) to −∞ before softmax, so those weights become 0. It prevents a token from attending to future tokens — required for autoregressive decoders like GPT. Turn it on in the tool to watch the upper-triangular weights collapse to zero.

Does this calculator support multi-head attention?

No. Version 1 is a single attention head with the Q, K, and V you supply — no learned Wᵩ/Wₖ/Wᵥ projections, no head concatenation, and no positional encodings. That keeps the arithmetic transparent for learning. To model multi-head attention, run one head at a time and concatenate the outputs yourself.

Can I use this to unit-test my own attention implementation?

Yes — that's a core use case. Enter the same small Q, K, V your code uses (up to 8×8), match the scaling and mask flags, and compare the raw scores, weights, and output against your implementation. The tool also cross-checks its stable softmax against the naive formula on every run and shows the difference, so you can trust the reference numbers.

AI · Transformers

Attention Score Calculator — Scaled Dot-Product Attention

Paste Query, Key, and Value matrices and see every step of a single attention head: the raw QKᵀ scores, the √dₖ-scaled scores, the softmax attention weights (each row summing to 1), and the output context vectors. Free, no signup, everything runs in your browser.

By Induwara Ashinsana— Executive Director, Ryzera TechnologiesUpdated Jul 18, 2026

Scaled dot-product attention

Query Q2×2

rows = query tokens, cols = dₖ

Key K2×2

same cols (dₖ) as Q

Value V2×2

same rows as K, cols = dᵥ

Scale by √dₖ· Vaswani §3.2.1Causal mask· decoder — hide future tokens

Examples

Shapes

2×2 · 2×2 · 2×2

Q · K · V

Head dim dₖ / dᵥ

2 / 2

scale √dₖ = 1.4142

Weight rows sum to 1

✓ yes

max error 0.0e+0

Stable vs naive softmax

Δ 0.0e+0

cross-check of the output

Decimals

1 · Raw scores

S = Q·Kᵀ

	K₁	K₂
Q₁	1.0000	0.0000
Q₂	0.0000	1.0000

2 · Scaled scores

S / √dₖ (√2 = 1.4142)

	K₁	K₂
Q₁	0.7071	0.0000
Q₂	0.0000	0.7071

3 · Attention weights

A = softmax(scores), row-wise

	K₁	K₂	Σ
Q₁	0.6698	0.3302	1.000
Q₂	0.3302	0.6698	1.000

4 · Output context vectors

O = A·V

	d₁	d₂
Q₁	1.6605	2.6605
Q₂	2.3395	3.3395

Trace one query row

Query Q₁ = [1.0000, 0.0000]
Scores (dot with each key, then ÷ √2 = 1.4142):
- Q₁·K₁ = 1.0000 → 0.7071
- Q₁·K₂ = 0.0000 → 0.0000
Softmax: subtract row max 0.7071, exponentiate, divide by Σ = 1.4931
- weight₁ = 1.0000 / 1.4931 = 0.6698
- weight₂ = 0.4931 / 1.4931 = 0.3302
Output O₁ = Σ weightⱼ · V_j = [1.6605, 2.6605]

Method: Attention(Q,K,V) = softmax(Q·Kᵀ / √dₖ)·V— Vaswani et al., “Attention Is All You Need” (2017), §3.2.1; decoder masking §3.2.3. Softmax is the stable subtract-the-max form (each row sums to 1). Everything runs in your browser — no data leaves this page.

How it works

This tool implements scaled dot-product attention exactly as defined in Vaswani et al., “Attention Is All You Need” (2017), §3.2.1:

Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ)·V

Given a query matrix Q (n×dₖ), a key matrix K (m×dₖ), and a value matrix V (m×dᵥ), the computation runs in four stages:

Raw scores. S = Q·Kᵀ, an n×m matrix. Each entry S_ij = Σ_p Q[i][p]·K[j][p] is the dot product of query i with key j — how strongly that query matches that key.
Scale.S′ = S / √dₖ. Dot products grow with the head dimension dₖ, and large values push softmax into flat regions where gradients vanish. Dividing by √dₖ (the standard deviation of a unit-variance dot product over dₖ terms) keeps the scores well-conditioned. If you turn scaling off, the tool uses the raw dot products directly.
Causal mask (optional).For decoder self-attention, entries with j > i are set to −∞ before softmax, so a token cannot attend to future positions (Vaswani §3.2.3). Those weights become exactly 0.
Softmax. Each row is normalised: A_ij = e^{S′_ij} / Σ_k e^{S′_ik}. The implementation subtracts the row maximum before exponentiating — a value-preserving trick that prevents floating-point overflow. Every row of A sums to 1, shown as a Σ column you can eyeball.
Output. O = A·V, an n×dᵥ matrix. Each output row O_i = Σ_j A_ij·V[j] is a weight-averaged blend of the value vectors — the context vector the attention head passes on.

To keep the reference numbers trustworthy, every run also recomputes the output with the naive softmax (no max-subtraction) and reports the largest difference between the two — normally around 1e-16 on ordinary inputs, and an overflow on extreme scores, which is exactly why the stable form is the default. All computation is pure JavaScript in the browser; nothing is uploaded.

Worked examples

Example 1 — one query, two keys (dₖ = 2, scaling on)

Inputs: Q = [1, 0], K = [[1,0],[0,1]], V = [[1,2],[3,4]]
Raw scores: Q·K₁ = 1·1 + 0·0 = 1 ; Q·K₂ = 1·0 + 0·1 = 0 → S = [1, 0]
Scale by √2 = 1.414214 → S' = [0.707107, 0]
Softmax: subtract max 0.707107 → exp([0, −0.707107]) = [1, 0.493069]
Σ = 1.493069 → weights = [0.669762, 0.330238] (sum = 1.000000 ✓)
Output: 0.669762·[1,2] + 0.330238·[3,4] = [1.660477, 2.660477]

Example 2 — causal mask on a 2-token sequence (dₖ = 2)

Inputs: Q = K = V = [[1,0],[0,1]], scaling on, causal mask on
Row Q₁ (i=0): key j=1 is in the future → score K₂ set to −∞
Scaled row = [0.707107, −∞] → softmax → weights [1, 0]
Output O₁ = 1·[1,0] = [1, 0]
Row Q₂ (i=1): nothing masked → scores [0,1] → weights [0.330238, 0.669762]
Output O₂ = 0.330238·[1,0] + 0.669762·[0,1] = [0.330238, 0.669762]

Example 3 — negative and fractional inputs (dₖ = 2, scaling on)

Inputs: Q = [−1, −2], K = [[0.5,0.5],[1,−1]], V = [[2,0],[0,2]]
Raw scores: Q·K₁ = −0.5 − 1 = −1.5 ; Q·K₂ = −1 + 2 = 1 → S = [−1.5, 1]
Scale by √2 → S' = [−1.060660, 0.707107]
Softmax: subtract max 0.707107 → exp([−1.767767, 0]) = [0.170711, 1]
Σ = 1.170711 → weights = [0.145821, 0.854179] (sum = 1.000000 ✓)
Output: 0.145821·[2,0] + 0.854179·[0,2] = [0.291642, 1.708358]

Frequently asked questions

Sources & references

The formula and worked examples on this page were last cross-checked against the sources above on 2026-07-18. The tool computes a single attention head only — no learned projections, positional encodings, or dropout.

Related tools

LiveAI

Softmax Calculator

Convert a list of logits into a probability distribution with the softmax function. Adjust temperature to flatten or sharpen the result, and see each class probability, the argmax, entropy and a numerically-stable step-by-step breakdown — verified against PyTorch, entirely in the browser.

Open tool

LiveAI

Davies-Bouldin Index Calc

Compute the Davies–Bouldin Index (DBI) of a clustering from raw data points and labels. Get the overall score, per-cluster dispersion and centroids, each cluster's worst rival Dᵢ, and the full pairwise Rᵢⱼ matrix. Matches scikit-learn davies_bouldin_score, runs entirely in your browser.

Open tool

LiveAI

Brier Score Calculator

Compute the Brier score and Brier Skill Score for probabilistic predictions. Paste forecast probabilities and 0/1 outcomes to get the mean-squared-error of the probabilities, the skill score versus a baseline, the exact formula and a per-pair breakdown. Matches scikit-learn, runs in the browser.

Open tool

Rate this tool

Be the first to rate

Comments & feedback

Spotted a bug or want an improvement? Tell us — our team reviews every comment, and good ideas get built. Comments are public and anonymous.

Found a bug, edge case, or want to suggest an improvement?

Email me at [email protected] — most fixes ship within 24 hours.