LLM KV Cache Memory Calculator

During generation a transformer keeps the Key and Value vectors of every past token so it does not recompute them each step. That store is the KV cache, and it is usually the memory that decides how long a context you can run and how many requests a GPU can serve at once. The size is exact arithmetic over the model's attention architecture — no training data or benchmarking needed.

The per-token size, summed across all layers and counting both the Key and the Value tensor, is:

per_token_bytes = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element

The leading 2is one tensor for Keys and one for Values. This is the formula given by Pope et al. (2022) in “Efficiently Scaling Transformer Inference” and mirrored in the Hugging Face Transformers KV-cache documentation. num_kv_headsis the grouped Key/Value head count from the model's config.json — for a grouped-query-attention model it is smaller than the number of attention heads, and that gap is exactly the memory it saves (Ainslie et al., 2023).

Two more multiplications finish the job, and the cache scales linearly in each:

• Per sequence = per-token bytes × context length. Doubling the context doubles the cache.
• Total = per-sequence bytes × batch size (concurrent sequences).

bytes_per_element is set by the cache precision: FP16/BF16 = 2, FP8 = 1, INT4 = 0.5. Halving the precision halves the cache, which is why FP8 KV cache is a common lever for longer context. All results use binary units (1 KiB = 2¹⁰, 1 GiB = 2³⁰ bytes) to match how allocators and nvidia-smireport memory. As a cross-check the tool also computes the cache the way vLLM's PagedAttention does — rounding each sequence up to whole 16-token blocks — which matches the closed-form figure exactly at block-aligned contexts.

When you enter a GPU's VRAM the tool switches on a capacity check. It converts the card's decimal GB to bytes, removes an overhead reserve of max(10%, 2 GiB) for activations, the CUDA context, and allocator fragmentation, then subtracts the model weights (parameter count × weight precision). Whatever is left is the KV budget: dividing it by the per-sequence cache gives the maximum concurrent requests, and dividing by the per-token cache gives the longest single-sequence context. The reserve is the one estimate here and is shown alongside the result; the vLLM PagedAttention paper (Kwon et al., 2023) documents why this cache budget, not raw compute, is what caps real serving concurrency. Sliding-window models are computed as a full-context upper bound, and DeepSeek-V3's multi-head latent attention uses its compressed-latent formula rather than the standard one.

• Pope et al. (2022) — Efficiently Scaling Transformer Inference (KV memory formula)
• Ainslie et al. (2023) — GQA: Training Generalized Multi-Query Transformer Models
• Kwon et al. (2023) — Efficient Memory Management for LLM Serving with PagedAttention (vLLM)
• Hugging Face Transformers — KV cache documentation (dtype & shape)
• Model config.json sources — Llama 3, Mistral, Qwen2.5, Gemma 2, DeepSeek-V3 (Hugging Face)

The formula and every model's architecture constants were last cross-checked against these sources on 2026-07-01. Architecture values are taken from each model's published config.json; the overhead reserve is an explicit, disclosed estimate.