Transformer Parameter Count Calculator

The calculator follows the closed-form parameter breakdown in EleutherAI's Transformer Math 101 and the original architecture papers. Write h for hidden size (d_model), L for the number of layers, V for vocabulary size, n_ctx for the maximum context length, and d_ff for the feed-forward inner size. Every parameter in a decoder-only transformer falls into one of three groups, embeddings, the repeated layer stack, and a single final norm, and the total is just their sum. Nothing is estimated: each term below is an exact integer.

Embeddings. The token embedding matrix holds V · h weights. Learned absolute positions add n_ctx · h; rotary (RoPE) and ALiBi positions are computed from head geometry rather than learned, so they add nothing, which is why modern long-context models can stretch their window without growing. (If you are extending a RoPE window, the RoPE context extension calculator covers the frequency-scaling side.) The output unembedding adds another V · h unless it is tied to the input embedding, in which case it costs nothing. Tying is the single biggest lever on small models: it is 31% of GPT-2 small.

Attention, per layer. Attention has four projections. With standard multi-head attention the query, key, value, and output matrices are each h×h, giving 4 · h² weights. Under grouped-query attention several query heads share one key/value head, so K and V shrink to h · (n_kv · head_dim), where head_dim = h / n_heads. Llama 3 8B pairs 32 query heads with 8 KV heads: its attention block is 41.9M parameters instead of the 67.1M that full MHA would cost, a 37% saving on the attention term and a 4× cut to KV cache memory at inference time. Head count must divide the hidden size evenly, and KV heads must divide the head count, the tool rejects configurations that do not.

Feed-forward, per layer. The classic block is two linear layers, so 2 · h · d_ff weights, with d_ff = 4 · h in the original paper and across the GPT-2 and GPT-3 family. That term alone is two thirds of a large GPT-style model. Gated MLPs, SwiGLU in Llama, Mistral, and Gemma, use three matrices (gate, up, down) of width d_ff, so they cost 1.5× the two-matrix figure. The tool models them by taking an effective d_ff of 1.5× the reported intermediate size: Llama 3's 14,336 becomes 21,504, which reproduces the published 8.03B exactly.

Norms and biases. Each layer carries two normalization layers. LayerNorm has a weight and a bias vector, so 4h per layer; RMSNorm drops the bias and the mean subtraction, so 2h. One final norm sits after the stack (2h or h). Bias vectors on the linear layers add 2h + 2·(n_kv · head_dim) for attention and d_ff + h for the feed-forward, per layer. These terms are linear in h while the matrices are quadratic, so they are negligible at scale, 49,152 of GPT-3's 1.81 billion per-layer parameters, about 0.003%, but they are exactly what makes a hand count miss the real number by a few thousand on a small model.

Totals and the scaling rule. The grand total is embeddings plus L times the per-layer count plus the final norm. For comparison, the calculator also prints the Kaplan et al. (2020) non-embedding approximation N ≈ 12 · L · h², which follows from assuming d_ff = 4h and multi-head attention: 4h² for attention plus 8h² for the feed-forward gives 12h² per layer. It holds to 0.14% on GPT-2 small and 0.01% on GPT-3. It reads 7.7% low on Llama 3 8B, and the reason is worth knowing: grouped-query attention makes that layer cheaper than 4h² (2.5h² here), but the SwiGLU MLP costs 10.5h² instead of 8h², so the two deviations do not cancel, the layer lands at 13h² and the rule undershoots. Change only the attention and it overshoots instead. The tool reports the signed deviation so you can see where your own configuration sits. Use the exact figure for memory planning and the approximation only for order-of-magnitude scaling arguments, such as the token budget in the Chinchilla compute-optimal calculator.

Where the parameters actually go: width, depth, and vocabulary

Three knobs decide almost everything, and they do not pull with equal force. Write r = d_ff / h for the feed-forward ratio and one layer's weights collapse to (4 + 2r) · h² under full multi-head attention, hidden size enters squared, layer count only linearly. Doubling h quadruples the stack; doubling L merely doubles it. GPT-2 Small to Medium is the clean demonstration: h rises 768 → 1,024 (1.33×) and L 12 → 24 (2×), and the layer stack grows from 85.1M to 302.3M, a factor of 3.55, which is 1.33² × 2 to three significant figures. Width is the expensive knob, which is why depth-vs-width tables in architecture papers look so lopsided.

Head count is free; KV head count is not. The head count has to divide h evenly, giving head_dim = h / n_heads, 64 in GPT-2 Small (768/12) and 128 across the Llama 3 family (4,096/32 and 8,192/64). Under full MHA that choice costs nothing: Q, K, V, and O stay h×h however many heads you slice them into, so re-splitting 32 heads into 64 leaves the parameter count untouched and only changes what each head attends to. The count moves the moment you cut KV heads. Llama 3 70B pairs 64 query heads with 8 KV heads, which takes attention from the 268.4M that 4h² would cost down to 151.0M per layer, a 43.8% cut on the attention term, compounded 80 times.

The vocabulary crossover. Embeddings are V · h, linear in width, so their share of the model collapses as the quadratic layer terms take over: 31.6% of GPT-2 Small, 14.8% of Medium, 8.5% of Large, 5.3% of XL, and 0.4% of GPT-3 175B. The trend is not monotonic across families, though, Llama 3 8B climbs back to 13.1%, because a 128,256-token vocabulary is 2.6× GPT-2's and the model unties its output head, so it carries two V · h matrices worth 1.05B parameters on a model only 8B in total. At that scale the tokenizer is an architecture decision, not a preprocessing detail; the tokenizer visualizer shows what a larger vocabulary buys you in tokens per word to set against that cost.

The feed-forward ratio, and why SwiGLU breaks the arithmetic. r = 4 runs from the 2017 paper through GPT-3, so the MLP is 8h² and two thirds of each layer. Gated models report a smaller intermediate size spread over three matrices: Llama 3 8B's 14,336 is 3.5h, but gate, up, and down together cost what a two-matrix block of 1.5 × 14,336 = 21,504 would, an effective r = 5.25, so the MLP is 10.5h², not 8h². Enter the reported number directly and you will undercount an 8B model by 1,879,048,192 parameters. That one substitution is the single most common reason a hand-rolled spreadsheet disagrees with a model card. Parameter-efficient fine-tuning changes the arithmetic again, since only a low-rank slice of these matrices is trainable, the LoRA calculator sizes that slice, and the training memory calculator turns either figure into optimizer and gradient footprint.

What the number excludes. A parameter count is weights only. It does not include optimizer state (Adam carries two moments, so training memory is several times the weights), gradients, activations, or the KV cache, and it is not the file size on disk until you multiply by bytes per parameter, 2 for FP16, 1 for INT8, about 0.5 for 4-bit quantization. The model size calculator does that conversion. Mixture-of-Experts models also break the single-number assumption, because total and active parameters differ by an order of magnitude; the MoE calculator handles that split. Everything on this page assumes a dense decoder-only stack.

Four configurations, each computed term by term. The first two are the canonical reference points, the third shows what grouped-query attention changes, and the fourth is the edge case where the familiar rule of thumb stops working.

The fourth example is the one worth keeping in mind. The 12·L·h² rule is a non-embedding approximation, and quoting it as a model's size is safe only once the layer stack dwarfs the vocabulary table. Below roughly a hundred million parameters, small language models, distilled students, classroom builds, the embedding matrix is the model, and a large tokenizer vocabulary costs more than adding layers. Shrinking the vocabulary or tying the output head buys far more than trimming d_ff at that scale.

Ten published architectures, with the exact count this calculator returns for each beside the figure its model card or paper reports. Nothing in the table is typed in by hand, every total is computed from the configuration in the same row when the page renders, so the table and the calculator can never disagree. Load any of the presets to walk a row line by line, or use it as a sanity check on a config you are entering yourself.

GPT-2 and GPT-3 rows use learned positions, LayerNorm, biases, and tied embeddings. Llama and Mistral rows use RoPE, RMSNorm, no biases, and untied embeddings, with d_ff set to 1.5× the model's reported intermediate size to stand in for the three-matrix SwiGLU MLP, 14,336 → 21,504 for Llama 3 8B, 28,672 → 43,008 for Llama 3 70B, 11,008 → 16,512 for Llama 2 7B. Totals round to the reported figure in every row.

Read down the last column and the scaling story is right there. Embeddings are 31.6% of GPT-2 Small and 0.4% of GPT-3 175B, because V · h grows linearly while the layer stack grows with h². Mistral 7B and Llama 3 8B share an identical layer stack, same hidden size, same depth, same 8 KV heads, same effective d_ff, so the same 6,979,588,096 non-embedding parameters, and differ by 788M purely because Llama 3 carries a 128,256-token vocabulary against Mistral's 32,000. Two models described as “7B” and “8B” are the same transformer with different tokenizers. If you need to turn any of these totals into a GPU shopping list, the GGUF quantization calculator covers what each quantization level costs in quality and bytes.

• EleutherAI: Transformer Math 101 (per-component parameter breakdown)
• Vaswani et al. 2017: Attention Is All You Need (transformer block, FFN = two linear layers)
• Radford et al. 2019: GPT-2 (architecture behind the 124M reconciliation)
• Brown et al. 2020: GPT-3 (Table 2.1, the 175B configuration)
• Kaplan et al. 2020: Scaling Laws (the 12·L·d_model² non-embedding approximation)
• Ainslie et al. 2023: GQA (grouped-query attention, the shrunken K/V projections)
• Shazeer 2020: GLU Variants Improve Transformer (SwiGLU, the three-matrix MLP)

Formulas and presets were last cross-checked against published parameter counts on 2026-10-02. Every built-in preset reproduces its model card's reported size.