How many warmup steps should I use for fine-tuning?

A common rule of thumb is 3–10% of total training steps, or a few hundred steps for short fine-tunes. Hugging Face defaults to 0, but most recipes set warmup_ratio around 0.03–0.06. For a 10,000-step run, 300–600 warmup steps is typical. Enter your total steps here and try a few values to see how the ramp changes.

What is the difference between linear and cosine learning rate decay?

Both ramp up during warmup and then decay to zero. Linear decay drops the LR in a straight line, so it falls fast early and reaches the floor at a steady pace. Cosine decay follows a half-cosine curve: it stays high longer, then drops steeply near the end. Cosine is the more common choice for large LLM training; linear is the transformers default for many fine-tunes.

How does get_cosine_schedule_with_warmup compute the learning rate?

During warmup the factor is step / warmup. After warmup it computes progress = (step − warmup) / (total − warmup) and returns 0.5 · (1 + cos(π · num_cycles · 2 · progress)), clamped at 0. With the default num_cycles = 0.5 this is a single half-cosine from the peak LR down to 0. The LR at each step is peak_lr times that factor.

What learning rate should the schedule end at — 0 or a floor?

The plain Hugging Face cosine and linear schedules decay to exactly 0. Many modern recipes instead keep a small floor (min LR), often 10% of the peak, so the model keeps learning slightly at the end. Set the Min LR field to a non-zero value to add a floor; the tool then uses lr = min_lr + (peak_lr − min_lr) · factor.

Do the numbers match what I see during real training?

Yes. The linear, cosine, and constant-with-warmup factors reproduce Hugging Face transformers' lr_lambda functions exactly, and the Noam curve uses the Vaswani formula verbatim. If your optimizer sets a base learning rate equal to the Peak LR you enter here, the per-step LR reported by the Trainer will match this tool step for step.

Does num_cycles above 0.5 make the LR oscillate?

Yes. num_cycles counts cosine half-waves across the decay phase. At 0.5 you get one smooth drop from peak to 0. At 1.0 the LR falls to 0 and rises back to the peak; at 1.5 it drops, rises, and drops again. Values above 0.5 are unusual for a single training run and are closer to warm-restart (SGDR) behaviour.

Is this tool specific to PyTorch or Hugging Face?

The factors follow the Hugging Face transformers definitions, which are the most widely referenced. They are framework-agnostic maths, so the same curve applies whether you use the transformers Trainer, raw PyTorch LambdaLR, or another library that implements the same warmup-then-decay shape. The generated snippet uses the transformers helper for convenience.

AI · Training

Learning Rate Scheduler Calculator

Plot the exact per-step learning rate for warmup, cosine, linear, constant, and Transformer (Noam) schedules — enter your peak LR, total steps, and warmup, then copy the matching Hugging Face code. Free, no signup, formulas cited below.

By Induwara Ashinsana— Executive Director, Ryzera TechnologiesUpdated Jul 8, 2026

Learning rate scheduleCosine with warmup

Matches Hugging Face transformers

Schedule type

Peak learning rate

The maximum LR reached at the end of warmup, e.g. 5e-4.

Total training steps

Total optimizer steps for the whole run.

Warmup steps

Steps to ramp LR up from 0 to peak.

Min LR (floor)

LR the schedule decays down to. 0 = full decay.

Cosine cycles

0.5 = one half-cosine (peak → floor). 1 = full wave.

Presets

Peak LR

0.0005

at step 100

LR at 50%

0.000293

step 500

Final LR

step 1,000

Peak : final

→ 0

decays to zero

LR vs. step

warmup ends at step 100

step 0peak 0.0005step 1,000

25% · step 250

0.000467

50% · step 500

0.000293

75% · step 750

8.93e-5

End · step 1,000

Code snippet

from transformers import get_cosine_schedule_with_warmup

# optimizer created with lr=0.0005 (your peak LR)
scheduler = get_cosine_schedule_with_warmup(
    optimizer,
    num_warmup_steps=100,
    num_training_steps=1000,
)

Sampled schedule

Step	Learning rate	Note
0	0
56	0.00028
100	0.0005	warmup end · peak
111	0.0005
167	0.000493
222	0.000478
278	0.000453
333	0.000422
389	0.000383
444	0.00034
500	0.000293
556	0.000245
611	0.000197
667	0.000151
722	0.000109
778	7.14e-5
833	4.13e-5
889	1.85e-5
944	4.76e-6
1,000	0	final

Factors reproduce the reference training-loop schedules.

How it works

A learning rate schedule multiplies your optimizer's learning rate (LR) by a factor that changes at every training step. Almost every modern transformer recipe follows the same two-part shape: a short warmup where the LR ramps up from 0 to the peak, then a decay back down. Warmup exists because a fresh model with random weights produces large, noisy gradients; starting at full LR can destabilise training, so the LR is eased in gradually (Goyal et al., 2017, §2.2).

For the linear, cosine, and constant schedules this tool reproduces the exact lr_lambda factors used by Hugging Face transformers, so the numbers match what the Trainer applies during real training:

Warmup (all schedules): factor = step / max(1, warmup), so the LR reaches the peak exactly at step = warmup.
Linear decay: factor = max(0, (total − step) / (total − warmup)) — a straight line from peak to 0.
Cosine decay: with progress = (step − warmup) / (total − warmup), factor = ½(1 + cos(π · num_cycles · 2 · progress)). At the default num_cycles = 0.5this is a single half-cosine from peak to 0, matching the SGDR annealing form of Loshchilov & Hutter (2017).
Constant with warmup: the LR holds at the peak after warmup (factor = 1).

You can add a floor with the Min LR field: the LR then follows lr = min_lr + (peak_lr − min_lr) · factor, so cosine and linear decay to your floor instead of 0. The Transformer (Noam) schedule is different — it is absolute rather than a multiple of a peak you choose. From Vaswani et al. (2017, §5.3): lr = d_model^(-0.5) · min(step^(-0.5), step · warmup^(-1.5)). It rises during warmup, peaks at step = warmup with value d_model^(-0.5) · warmup^(-0.5), then decays as the inverse square root of the step. The tool derives and shows that computed peak so you can compare schedules on the same axis.

Worked examples

Cosine with warmup

peak 5e-4 · total 1000 · warmup 100 · floor 0

Step 50 (warmup): factor = 50 / 100 = 0.5 → lr = 5e-4 × 0.5 = 2.5e-4
Step 100 (peak): progress = 0, factor = ½(1 + cos 0) = 1 → lr = 5e-4
Step 550: progress = (550 − 100) / 900 = 0.5
factor = ½(1 + cos(π × 0.5)) = ½(1 + 0) = 0.5 → lr = 2.5e-4
Step 1000: progress = 1, factor = ½(1 + cos π) = 0 → lr = 0

Linear with warmup

peak 2e-5 · total 10,000 · warmup 500 · floor 0

Step 250 (warmup): factor = 250 / 500 = 0.5 → lr = 1e-5
Step 500 (peak): factor = 9500 / 9500 = 1 → lr = 2e-5
Step 5250: factor = (10000 − 5250) / 9500 = 4750 / 9500 = 0.5 → lr = 1e-5
Step 10000: factor = 0 / 9500 = 0 → lr = 0

Transformer (Noam) — edge of decay

d_model 512 · warmup 4000 (paper base config)

d_model^(-0.5) = 1 / √512 = 0.044194
Peak at step 4000: 0.044194 × 4000^(-0.5) = 0.044194 × 0.0158114 = 6.99e-4
Step 1000 (warmup): 0.044194 × 1000 × 4000^(-1.5) = 0.044194 × 3.953e-3 = 1.75e-4
Step 16000 (= 4× warmup): 0.044194 × 16000^(-0.5) = 0.044194 × 0.0079057 = 3.49e-4
Note 16000 is 4× the peak step, and the LR is ≈ half the peak — the inverse-sqrt signature.

Frequently asked questions

Sources & references

The factors on this page were last cross-checked against the Hugging Face transformers source and the Vaswani formula on 2026-07-08. They are reviewed whenever the reference schedule implementations change.

Related tools

LiveAI

Effective Batch Size Calc

Compute the effective (global) batch size of an LLM fine-tune from per-device batch, gradient accumulation steps, and GPU count — plus optimizer steps, warmup steps, and tokens seen. Reverse-solves the accumulation steps for a target batch. HF Trainer formula cited, runs in your browser.

Open tool

LiveAI

AI Training Steps Calc

Compute optimizer steps per epoch, total training steps for max_steps, global batch size, forward/backward passes, approximate tokens seen and warmup steps from your dataset size, per-device batch, GPU count, gradient accumulation and epochs — counted exactly the way the Hugging Face Trainer counts them, with drop_last handling and a copy-ready TrainingArguments snippet. Runs entirely in your browser.

Open tool

LiveAI

Cosine Similarity Calc

Compute the cosine similarity, cosine distance, and angle between two numeric vectors or two short texts, with the full dot-product and magnitude working. Matches scikit-learn, runs entirely in the browser.

Open tool

Rate this tool

Be the first to rate

Comments & feedback

Spotted a bug or want an improvement? Tell us — our team reviews every comment, and good ideas get built. Comments are public and anonymous.

Found a bug, edge case, or want another schedule added?

Email me at [email protected] — most fixes ship within 24 hours.

LR vs. step

Code snippet

Sampled schedule

How it works

Worked examples

Frequently asked questions

How many warmup steps should I use for fine-tuning?

What is the difference between linear and cosine learning rate decay?

How does get_cosine_schedule_with_warmup compute the learning rate?

What is the Transformer (Noam) learning rate schedule formula?

What learning rate should the schedule end at — 0 or a floor?

Do the numbers match what I see during real training?

Does num_cycles above 0.5 make the LR oscillate?

Is this tool specific to PyTorch or Hugging Face?

Sources & references

Related tools

Effective Batch Size Calc

AI Training Steps Calc

Cosine Similarity Calc

Comments & feedback