Speculative Decoding Speedup Calculator

Speculative decoding speeds up large-language-model inference by letting a small, cheap draft model guess several tokens ahead, then verifying all of them in a single forward pass of the large target model. Any prefix of guesses the target agrees with is kept for free; the first disagreement is corrected by the target itself. Because the expensive model runs once per batch of γ guesses instead of once per token, wall-clock latency drops, without changing the output.

This tool implements the exact improvement-factor model from Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman & Matias, ICML 2023). Every quantity is measured in units of one target-model forward pass, so the result is a hardware-independent multiplier. Three inputs drive it:

• Acceptance rate α, the probability the target accepts a given draft token (an empirical number you measure on your own prompts).
• Draft tokens per step γ, how many tokens the draft speculates before each verification.
• Cost ratio c, one draft forward pass divided by one target forward pass; estimate it as target tokens/s ÷ draft tokens/s.

The math is three deterministic steps:

1. Expected tokens per step. The number of tokens produced per iteration is a geometric series over accepted draft tokens plus the one guaranteed target token: E = (1 − α^(γ+1)) / (1 − α), which equals 1 + α + α² + … + α^γ. The calculator computes it both ways as a self-check.
2. Cost per step. The draft runs γ times at cost c each, plus exactly one parallel target verification pass: cost = γ·c + 1.
3. Speedup (improvement factor). Expected tokens divided by cost per step (Theorem 3.8): speedup = E / (γ·c + 1). A value above 1 is a real wall-clock win; below 1 means the overhead outweighs the savings.

Speedup is concave in γ: it rises, peaks at an interior optimum, then falls as each extra draft token costs more than it earns. That optimum shrinks as the draft gets slower (larger c). The tool scans γ = 1 to 12 and reports the peak, so you can pick the best draft length directly. Finally, per Chen et al. (2023), the accept/reject scheme is distribution-preserving, the output is identical to plain target sampling, so this is purely a latency gain.

• Leviathan, Kalman & Matias: Fast Inference from Transformers via Speculative Decoding (ICML 2023)
• Chen et al. (DeepMind): Accelerating LLM Decoding with Speculative Sampling
• Hugging Face: Assisted Generation: low-latency text generation

The speedup and expected-tokens formulas were last cross-checked against Leviathan et al., ICML 2023 (arXiv:2211.17192) on 2026-07-08. The two worked examples above (2.40× and 1.36×) are reproduced exactly by the calculator.