LLM Training Compute (FLOPs) Calculator

Every figure comes from two published scaling-law results, applied as closed-form arithmetic in your browser — nothing is trained, uploaded, or downloaded.

1. Total compute — the 6ND rule

Training a dense transformer costs about six floating-point operations per parameter per token: C ≈ 6 × N × D, where N is the parameter count and D is the number of training tokens. The factor 6 is two FLOPs for the forward pass plus four for the backward pass (the backward pass costs roughly twice the forward pass). This is the standard estimate from Kaplan et al.'s Scaling Laws for Neural Language Models (2020).

2. PetaFLOP/s-days

To compare runs, total FLOPs are divided by the compute in one petaFLOP/s-day: 1 PF/s-day = 10¹⁵ × 86,400 = 8.64×10¹⁹ FLOPs. This is the unit OpenAI used to report GPT-3 at ≈3,640 petaFLOP/s-days.

3. Chinchilla-optimal tokens

Hoffmann et al. (2022) showed that for a fixed compute budget, model size and dataset size should grow together — about D ≈ 20 × N training tokens per parameter. The tool computes that optimum and flags your run as undertrained (well below it), compute-optimal (within ±25%), or over-trained (well above it — a deliberate choice when you want a smaller model that is cheaper to serve, as with Llama-2). The ±25% band is deliberate: Hoffmann et al. report a flat loss basin around the optimum, so a run does not have to hit 20 tokens per parameter exactly to be treated as compute-optimal. If you want to work the problem the other way — start from a fixed FLOP budget and split it into the best N and D — use the compute-optimal (Chinchilla) calculator, which inverts the same two equations.

4. GPU-hours, wall-clock, and cost

GPU-hours are the compute divided by the GPU's effective throughput: GPU-hours = C / (peak FLOP/s × MFU) / 3600. Peak throughput is the dense BF16/FP16 tensor figure from the NVIDIA datasheet (A100 312 TFLOP/s, H100 SXM 989 TFLOP/s), and MFU (Model FLOPs Utilization) is the fraction of peak a real run achieves — typically 30–55%. Wall-clock time divides GPU-hours by the number of GPUs (ideal scaling), and cost multiplies GPU-hours by your price per GPU-hour. MFU is the single most sensitive input here: GPU-hours are inversely proportional to it, so a run planned at 50% MFU that actually lands at 25% costs exactly twice as much. If you are shopping for a price per GPU-hour rather than guessing one, the GPU cloud cost calculator compares RunPod, Lambda, Vast.ai, and AWS rates for the same card.

5. Cross-check against published runs

As an independent check, the 6ND formula is applied to three documented runs. For GPT-3 and Chinchilla — where the papers state a compute number — the tool shows the percentage difference. The 6ND estimate reproduces GPT-3's published 3.14×10²³ FLOPs to within about 0.3%, the same idea as cross-checking a tax figure against the regulator's own formula. This is the dense approximation: Mixture-of-Experts, the attention-FLOPs correction, and very long context can shift the real number by ±10–30%.

6. Guard rails and edge cases

Inputs are bounded before anything is computed: 1K to 100T parameters, 1K to 1,000T training tokens, and MFU between 5% and 80%. A blank, zero, negative, or non-numeric entry is clamped to zero rather than producing NaN, so the page never shows a broken figure — you get a specific message telling you which field to fix. A zero or missing GPU peak yields zero GPU-hours instead of dividing by zero, and the Chinchilla ratio falls back to “undertrained” when the parameter count is zero. Every number is recomputed synchronously in the browser as you type; nothing is sent to a server.

6ND is an approximation, and knowing where it drifts is the difference between a planning figure and a wrong budget. Four corrections matter in practice.

Attention FLOPs at long context

The 6ND term counts only the matrix multiplies whose cost scales with parameters. The attention score computation scales with sequence length instead, adding roughly 12 × L × s × d FLOPs per token, where L is the layer count, s the sequence length, and d the model dimension. Kaplan et al. note this stays a small fraction of the total while d > s / 12. For Llama-2 7B (L = 32, d = 4,096, s = 4,096) the correction is 12 × 32 × 4,096 × 4,096 ≈ 6.4×10⁹ FLOPs per token against 6N = 4.2×10¹⁰ — about 15% on top, and roughly half that once causal masking is accounted for. Push the same model to 32K context and attention stops being a rounding error. If context length is the variable you are actually reasoning about, the context window calculator is the better starting point.

Activation recomputation

Large runs trade compute for memory with gradient checkpointing, re-running the forward pass during the backward pass to avoid storing activations. Full recomputation turns 6ND into roughly 8ND — a flat 33% surcharge that this tool does not add. By convention, Model FLOPs Utilization deliberately excludes recompute (that is what separates MFU from Hardware FLOPs Utilization), so an MFU figure quoted from a paper is already the right thing to divide by here. Do not add the 33% and lower the MFU; that double-counts.

Mixture-of-Experts

For an MoE model, N in 6ND is the parameters activated per token, not the total checkpoint size. Mixtral 8×7B holds about 46.7B parameters but routes roughly 12.9B per token, so using the total overstates training compute by about 3.6×. The worked example below shows the same mistake at GPT-4 scale. Total parameters still govern memory, which is what the LLM VRAM calculator sizes.

Embeddings and the small-model regime

Kaplan et al. fit their scaling laws on non-embedding parameters. For a 70B model the embedding and output matrices are a rounding error, but for a 100M-parameter model with a 128K vocabulary they can be a third of the checkpoint — so a small-model estimate that includes embeddings in N runs high. Beyond that, restarts, failed runs, checkpoint reloads, and evaluation passes are real spend that no closed-form formula captures. Budget 10–20% headroom on top of whatever this page returns.

Total FLOPs is the physics of the run — hardware-independent, and the number to quote when comparing your plan against a published model. PetaFLOP/s-days is the same quantity in the unit the GPT-3 paper used, so it is the easiest column to eyeball against the reference table.

GPU-hours and costare the first hardware-dependent figures, and both move with MFU and card choice. Quoted cloud rates usually assume on-demand pricing; spot and reserved capacity can be 40–70% cheaper but expose you to preemption, which means checkpointing overhead. If you are budgeting in rupees rather than dollars, convert the USD figure at the day's rate with the currency converter — GPU rental is invoiced in USD everywhere, so the rupee cost of a fixed run moves with the exchange rate, not with anything about the model.

Wall-clock time assumes perfect linear scaling and is therefore a lower bound. Real clusters lose 10–40% to gradient all-reduce, pipeline bubbles, stragglers, and restarts, and that loss grows with node count — 512 GPUs will not be exactly 64× faster than eight. Two things the compute figure does not tell you: whether the model plus optimizer states even fit in the memory you have, and what the run costs in electricity and emissions. Those are the VRAM calculator and the AI energy & carbon calculator respectively. And if the goal is adapting an existing model rather than pre-training one, the fine-tuning cost calculator works at the scale you actually care about — fine-tuning typically costs three to four orders of magnitude less than the pre-training run modelled here.

The fastest way to sanity-check a compute estimate is to run the same formula over models whose training cost is a matter of public record. Below, the FLOPs and petaFLOP/s-days columns are what 6ND returns from the parameter and token counts in the same row — nothing in those two columns is quoted from a paper. The last column is what the source actually published, so the gap between them is the error of the approximation itself.

Three things fall out of that table. First, where a paper states a compute number at all, 6ND lands within about 2% of it — GPT-3 to 0.3%, PaLM to 1.3%, Llama 3.1 405B to under 0.2%. For dense models the approximation is far tighter than the ±10–30% caveat elsewhere on this page suggests; that wider band is for the cases where architecture or context length pulls the run away from the assumptions, not for a standard dense pre-train.

Second, the implied MFU column is the real spread. Divide published GPU-hours into the 6ND compute and the frontier runs cluster between 35% and 46% — PaLM at 46.2% on TPU v4, Llama-2 around 41–43% on A100, Llama 3.1 405B and DeepSeek-V3 both near 35% on Hopper-class hardware. The pattern is that utilization falls as cluster size rises, because communication and stragglers eat more of the budget. If you are planning a run on more than a few hundred GPUs, 35% is a more defensible default than the 40% the calculator starts with, and the GPU cloud cost calculator will show what that five-point difference costs at your provider's rate.

Third, DeepSeek-V3 is the cleanest published demonstration of the MoE trap. Its checkpoint holds 671B parameters but routes only 37B per token, so 6ND on the total would return 5.96×10²⁵ FLOPs against a true 3.29×10²⁴ — an 18× overstatement, and a budget wrong by more than an order of magnitude. Enter active parameters here. The total is still the number that governs how much memory the weights occupy, which is what the LLM VRAM calculator sizes, and the number that decides whether you can serve the model at all once training finishes.

A note on reading these as costs. Published GPU-hours are hardware time, not invoices — Meta and Google train on owned fleets, so no rental rate applies. Multiplying Llama 3.1 405B's 30.84M H100-hours by a $2.50 on-demand rate gives roughly $77M, which is what the run would have cost a renter, not what it cost Meta. The tool prices your plan the renter's way because that is the decision most people using this page are actually making.

• Kaplan et al. (2020) — Scaling Laws for Neural Language Models (the 6ND rule)
• Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models (Chinchilla, 20 tokens/param)
• Brown et al. (2020) — Language Models are Few-Shot Learners (GPT-3 compute, Table D.1)
• NVIDIA — H100 datasheet (peak BF16/FP16 tensor throughput)
• NVIDIA — A100 datasheet (peak BF16/FP16 tensor throughput)
• Chowdhery et al. (2022) — PaLM (the MFU definition and a reported 46.2% MFU)
• Korthikanti et al. (2022) — Reducing Activation Recomputation (MFU vs. HFU, the 8ND surcharge)
• Jiang et al. (2024) — Mixtral of Experts (46.7B total vs. 12.9B active parameters)
• Touvron et al. (2023) — Llama 2 (Table 2: pre-training A100-80GB hours per model size)
• Grattafiori et al. (2024) — The Llama 3 Herd of Models (405B: 3.8×10²⁵ FLOPs, 15.6T tokens, 30.84M H100-hours)
• DeepSeek-AI (2024) — DeepSeek-V3 Technical Report (37B active of 671B, 14.8T tokens, 2.664M H800-hours)

Formulas, constants, and GPU peak-throughput figures were last verified against the sources above on 2026-06-09; the published-runs table was compiled separately on 4 August 2026. The output is an order-of-magnitude planning estimate, not a guarantee — real runs vary ±10–30% with architecture, context length, and achieved MFU. The GPT-4-class figures used in the worked example and FAQ are third-party estimates, not OpenAI-published numbers, and are marked as such wherever they appear.