Adam Optimizer Calculator — step-by-step update rule

This tool implements the Adam optimizer exactly as defined in Kingma & Ba, “Adam: A Method for Stochastic Optimization” (ICLR 2015), Algorithm 1. Adam keeps two running averages per parameter — a first moment (the mean of recent gradients) and a second moment (the mean of recent squared gradients) — and uses them to give every parameter its own adaptive step size.

The state starts at m₀ = 0 and v₀ = 0. For each step t = 1, 2, … with gradient gₜ, five lines run in order:

1. First moment. mₜ = β1·mₜ₋₁ + (1−β1)·gₜ — an exponential moving average of the gradient. With β1 = 0.9 it blends 90% of the old estimate with 10% of the new gradient.
2. Second moment. vₜ = β2·vₜ₋₁ + (1−β2)·gₜ² — the same idea for the squared gradient, an estimate of its uncentred variance. β2 = 0.999 makes it change slowly.
3. Bias correction. m̂ₜ = mₜ/(1−β1ᵗ) and v̂ₜ = vₜ/(1−β2ᵗ). Because m and v start at 0 they are biased toward 0 early on; dividing by 1 − βᵗ counteracts that. The factor is largest at t = 1 and tends to 1 as t grows.
4. Parameter update. θₜ = θₜ₋₁ − α·m̂ₜ/(√v̂ₜ + ε). The direction comes from m̂ₜ, the per-parameter scaling from √v̂ₜ, and ε guards the division. At t = 1 with zero initial state this reduces to a step of about ±α regardless of the gradient magnitude.

The calculator mirrors the update ordering and defaults of PyTorch's torch.optim.Adam and Keras' tf.keras.optimizers.Adam (with ε = 1e-8, no weight decay, no AMSGrad). To keep the numbers trustworthy, every step is also computed with the paper's equivalent “efficient” reformulation — αₜ = α·√(1−β2ᵗ)/(1−β1ᵗ) applied to the raw moments — and the two results are compared on screen. They agree to floating-point precision. All computation is pure JavaScript in your browser; nothing is uploaded.

• Kingma & Ba, “Adam: A Method for Stochastic Optimization” (ICLR 2015) — Algorithm 1 and recommended defaults
• PyTorch — torch.optim.Adam reference (update ordering + defaults)
• TensorFlow / Keras — tf.keras.optimizers.Adam reference

The formulas, defaults, and worked examples on this page were last cross-checked against the sources above on 2026-07-18. The tool models vanilla Adam for a single scalar parameter — no AdamW/weight decay, AMSGrad, or learning-rate schedules.