LLM Training Memory (VRAM) Calculator

Training memory is dominated by four parts. This calculator computes each in bytes from published transformer formulas and shows the result in GiB (1 GiB = 2³⁰ bytes), the unit GPU VRAM is actually sized in.

Model states — weights, gradients, optimizer. The ZeRO paper models these as a fixed number of bytes per parameter. For mixed-precision AdamW that is 2 + 2 + 12 = 16 bytes/param: a 2-byte FP16 weight, a 2-byte FP16 gradient, and 12 bytes of Adam state (a 4-byte FP32 master copy plus the first and second moments, m and v, at 4 bytes each). Full FP32 AdamW is also 16 bytes/param (4 + 4 + 8); 8-bit Adam compresses the moments to 10 bytes/param. Multiply by the parameter count Ψ and you have the model state: K · Ψ.

Activations. These scale with sequence length and batch, not parameter count, and often dominate for smaller models. Using the per-layer formula from Korthikanti et al. (no recomputation):

act_per_layer = s · b · h · (34 + 5 · a · s / h)

where s is sequence length, b the micro-batch, h the hidden size, and a the number of attention heads. Total activations are that times the number of layers L. With gradient (activation) checkpointing, only one input tensor per layer is stored and the rest are recomputed in the backward pass, cutting activations to about L · 2 · s · b · h bytes.

Sharding across GPUs. With plain data-parallel training the full model state sits on every GPU. ZeRO partitions it instead: Stage 1 shards the optimizer states, Stage 2 also the gradients, and Stage 3 (equivalent to PyTorch FSDP) the weights too — so across N GPUs Stage 3 divides the whole 16-bytes/param state by N. Activations are never sharded by ZeRO; they stay per-GPU. The single-device total is the sum of the four parts; the per-GPU figure applies the sharding you choose. Add roughly 1 GiB for the CUDA context and framework buffers on top.

• EleutherAI — Transformer Math 101 (bytes-per-parameter model + activations)
• Rajbhandari et al. 2020 — ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Korthikanti et al. 2022 — Reducing Activation Recomputation in Large Transformer Models
• Hugging Face — Model training anatomy (byte breakdown + 8-bit optimizer)

Formulas and byte-per-parameter figures on this page were last cross-checked against the sources above on 2026-07-18. Estimates are close but not exact — framework buffers, attention kernels, and fragmentation move real usage by 10–20%.