How do I calculate effective batch size with gradient accumulation?

Multiply three numbers: per-device batch size × gradient accumulation steps × number of GPUs. That product is the effective (global) batch size — the number of examples that contribute to one optimizer update. Gradient accumulation runs several forward/backward passes and only steps the optimizer once, so it raises the effective batch without needing more VRAM.

What gradient accumulation steps do I need for a batch size of 128?

Divide the target by (per-device batch × GPUs) and round up. On one GPU with a per-device batch of 2 you need ceil(128 / 2) = 64 accumulation steps. With 4 GPUs and a per-device batch of 4 you need ceil(128 / 16) = 8. Switch this tool to 'Solve accum' mode and it does the division for you.

How many optimizer steps are in one epoch?

One epoch is the dataset size divided by the effective batch size, rounded up: ceil(D / B). For 50,000 examples and an effective batch of 64 that is ceil(781.25) = 782 optimizer steps per epoch. If you drop the last incomplete batch it rounds down instead: floor(781.25) = 781.

Does gradient accumulation change the effective batch size?

Yes — that is its whole purpose. Accumulating gradients over N micro-batches before one optimizer step makes the update behave as if the batch were N times larger. It does not change how much memory a single micro-batch uses, which is why it lets a small GPU train at a large effective batch, just more slowly.

How many tokens does my fine-tune see per step?

Tokens per optimizer step = effective batch size × sequence length. An effective batch of 64 at 1,024 tokens per example is 65,536 tokens per step. Across the whole run it is that figure times the total number of optimizer steps, which this tool reports alongside the naive dataset × length × epochs estimate.

Why does the total-token count differ from dataset × length × epochs?

Because the last batch of each epoch is usually padded up to a full effective batch rather than left partial. That padding adds a few tokens per epoch, so the step-based total sits slightly above the naive product. Turn on 'drop the last incomplete batch' and the two converge, at the cost of skipping a few examples each epoch.

Is the effective batch size the same across single-GPU and multi-GPU runs?

Only if you keep the product constant. With data parallelism each of the w GPUs processes its own micro-batch, so the world size multiplies the effective batch. Moving from 1 to 2 GPUs at the same per-device batch and accumulation doubles the effective batch — halve the accumulation steps to keep it fixed.

Does this tool download or run a model?

No. Every number is integer arithmetic computed in your browser from the batch, accumulation, GPU, dataset, and epoch values you type. Nothing is uploaded and no model weights are touched, so you can size warmup and estimate steps before renting any GPU time.

AI · Fine-tuning

Effective Batch Size & Gradient Accumulation Calculator

Find the effective (global) batch size of an LLM fine-tune from your per-device batch, gradient accumulation steps, and GPU count — then get optimizer steps, warmup steps, and tokens seen. Reverse mode solves the accumulation steps for a target batch. Runs in your browser, formula cited.

By Induwara Ashinsana— Executive Director, Ryzera TechnologiesUpdated Jul 4, 2026

Effective batch size

Formula cited · HF Trainer

Mode

Popular setups

Per-device batch size (m)

per_device_train_batch_size — what fits in VRAM.

Gradient accumulation steps (g)

gradient_accumulation_steps.

Number of GPUs (w)

World size — data-parallel devices.

Dataset size (examples)

Epochs

Sequence length (tokens)

Warmup ratio

warmup_ratio × total steps = warmup steps.

Last batch each epoch

Effective batch size

m × g × w = 4 × 8 × 1

Micro-batches / step

Optimizer steps / epoch

1,563

Total optimizer steps

4,689

Warmup steps

141

Tokens / optimizer step

65,536

Total tokens seen

307,298,304

Step-by-step

Quantity	Formula	Value
Effective batch (B)	m × g × w	32
Micro-batches / step	g × w	8
Steps / epoch	ceil(D / B)	1,563
Total steps (S)	steps/epoch × E	4,689
Warmup steps	round(S × ratio)	141
Tokens / step	B × L	65,536
Total tokens	S × B × L	307,298,304

Cross-check: the naive token count D × L × E is 307,200,000. Using HF Trainer's internal dataloader-floor method the run is 4,686 optimizer steps — within a few of the 4,689 headline, the difference being how the final partial batch of each epoch is counted.

Paste into your trainer

TrainingArguments(
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    num_train_epochs=3,
    warmup_ratio=0.03,
)  # effective batch = 4 × 8 = 32

Effective-batch and warmup formulas follow the Hugging Face TrainingArguments reference and the gradient-accumulation guide. Every figure is integer arithmetic computed in your browser — no model, no upload, no API call.

How it works

Every figure is integer arithmetic — no model is downloaded or run. The calculator uses the same definition the Hugging Face Trainer applies internally. Let m be the per-device batch size, g the gradient accumulation steps, w the world size (number of GPUs), D the dataset size, E the epochs, L the sequence length, and r the warmup ratio.

1. Effective (global) batch size

B = m × g × w. Gradient accumulation runs g forward/backward passes on each device before a single synchronized optimizer update, so it simulates a larger batch without extra memory. Data parallelism across w GPUs multiplies it again. (Hugging Face, TrainingArguments.)

2. Optimizer steps per epoch

With the last partial batch kept and padded, steps/epoch = ceil(D / B); with drop last on it is floor(D / B). Total optimizer steps are S = steps/epoch × E, rounded to an integer for fractional epochs. HF's Trainer floor-divides the dataloader length by the accumulation steps, which can shift the count by a step or two at each epoch boundary — the tool shows both so the gap is explicit rather than hidden.

3. Warmup steps

warmup = round(S × r). This mirrors how the Trainer derives warmup_steps from warmup_ratio when you do not set an absolute step count — useful for a linear or cosine schedule that ramps the learning rate over the first few percent of training.

4. Tokens seen

Tokens per optimizer step are B × L, and the run total is S × B × L. That sits a little above the naive D × L × E because the padded final batch of each epoch adds tokens; dropping the last batch removes the gap.

5. Reverse mode

Given a target effective batch B*, the accumulation steps are g = ceil(B* / (m × w)). When B* is not divisible by m × w the achieved batch overshoots — the tool reports the real batch and how far above target it lands, because you can only accumulate a whole number of steps.

Worked examples

Forward — 2×GPU QLoRA, effective batch 64

Inputs: m=4, g=8, w=2, D=50,000, E=3, L=1,024, warmup 0.03
Effective batch B = 4 × 8 × 2 = 64
Micro-batches / step = g × w = 8 × 2 = 16
Steps / epoch = ceil(50,000 / 64) = ceil(781.25) = 782
Total steps S = 782 × 3 = 2,346
Warmup = round(2,346 × 0.03) = round(70.38) = 70
Tokens / step = 64 × 1,024 = 65,536
Total tokens = 2,346 × 65,536 = 153,747,456 (naive 153,600,000)

Reverse — solve accumulation for batch 128 on one GPU

Inputs: target B* = 128, m = 2, w = 1
g = ceil(128 / (2 × 1)) = ceil(64) = 64
Achieved batch = 2 × 64 × 1 = 128 — exact, no overshoot

Edge case — reverse target that is not divisible

Inputs: target B* = 100, m = 3, w = 1
g = ceil(100 / 3) = ceil(33.33) = 34
Achieved batch = 3 × 34 = 102
Overshoot = 102 − 100 = 2 (you cannot accumulate a fraction of a step)

Frequently asked questions

Sources & references

The effective-batch, optimizer-step, and warmup formulas were last cross-checked against the Hugging Face Trainer documentation on 2026-07-04. The page is reviewed when the Trainer batch or scheduler semantics change.

Related tools

LiveAI

AI Audio Token Cost Calc

Convert an audio clip's duration (or a measured audio_tokens count) into the exact audio input tokens GPT-4o-audio and Gemini bill, then price it per request and per month in USD and LKR. Gemini's fixed 32 tokens/second rule is cited; compares all four models side by side. Runs in your browser, no signup.

Open tool

LiveAI

RAG Cost Calculator

Estimate the full one-time and monthly cost of a Retrieval-Augmented Generation pipeline — embedding, vector storage, retrieval, and LLM generation — across OpenAI, Anthropic, Google, Cohere and Pinecone, in USD and LKR. Every price sourced.

Open tool

LiveAI

Fine-Tuning Cost Calculator

Estimate the one-time training cost and ongoing monthly inference cost of fine-tuning an LLM (GPT-4o, GPT-4o mini, GPT-3.5 Turbo, Together AI Llama) from your token count, epochs, and usage — in USD and LKR.

Open tool

Rate this tool

Be the first to rate

Comments & feedback

Spotted a bug or want an improvement? Tell us — our team reviews every comment, and good ideas get built. Comments are public and anonymous.

Found a bug, edge case, or want another training-config field added?

Email me at [email protected] — most fixes ship within 24 hours.