RoPE Scaling Calculator — Extend LLM Context Length

Rotary Position Embedding (RoPE) encodes token position by rotating each query/key pair at a per-dimension frequency θ_i = θ^(−2i/d) for i = 0 … d/2−1, where θ is the base (rope_theta) and d is the head dimension (Su et al., 2021). A model only learns rotation angles up to the positions it saw in training. To read longer sequences you keep those long-position angles inside the trained range by rescaling RoPE.

The extension factor is the whole story's starting point: s = L_target / L_train. The four methods differ in how they apply it:

1. Linear Position Interpolation. Divide every position index by s before applying RoPE, so position L_target lands on the angle of L_train. Config {"type":"linear","factor":s}. It stretches all frequencies equally, blurring short-range detail, so it almost always needs fine-tuning at s ≥ 2 (Chen et al., 2023).
2. NTK-aware scaling. Instead of scaling positions, scale the base so high-frequency (short-range) dimensions are barely touched while low-frequency (long-range) dimensions interpolate: θ′ = θ · s^(d/(d−2)). Set the model's rope_theta to θ′. This often gives usable zero-shot extension up to ~2× (bloc97 / kaiokendev).
3. Dynamic NTK.The same θ′ formula, but the scale is recomputed as the sequence grows past L_train, so short prompts run at the original base and are not degraded. Config {"type":"dynamic","factor":s}.
4. YaRN. NTK-by-parts — interpolate low-frequency dimensions, extrapolate high-frequency ones, with boundary wavelengths set by α=1, β=32 for Llama-class models — plus an attention-softmax temperature. The recommended temperature satisfies √(1/t) = 0.1·ln(s) + 1, i.e. logits are multiplied by m = 0.1·ln(s) + 1. Config {"type":"yarn","factor":s,"original_max_position_embeddings":L_train}. It holds quality best at large s (Peng et al., 2023).

Every output is a deterministic function of the four inputs — no data source drift, no network call. As a self-check, the NTK-aware base is derived a second way: requiring the lowest-frequency dimension pair (i = d/2−1) to have its wavelength stretched by exactly s and solving for the new base gives the same θ′ to the integer. The accepted Hugging Face rope_scaling types are linear, dynamic and yarn; static NTK is applied by overwriting rope_theta directly.

• Su et al. — RoFormer: Enhanced Transformer with Rotary Position Embedding (arXiv:2104.09864)
• Chen et al. — Extending Context Window of LLMs via Position Interpolation (arXiv:2306.15595)
• Peng et al. — YaRN: Efficient Context Window Extension of LLMs (arXiv:2309.00071)
• bloc97 / kaiokendev — NTK-Aware Scaled RoPE
• Hugging Face Transformers — LlamaConfig rope_scaling field

The RoPE, Position Interpolation, NTK and YaRN formulas were last cross-checked against the arXiv sources above on 2026-07-10. The NTK-aware base is computed in double precision and verified a second way from the lowest-frequency wavelength, so both derivations agree to the integer.