LLM Concurrency & GPU Replica Calculator

The calculator turns four measured numbers — per-stream decode speed s (tokens/second), batch slots B(the maximum concurrent sequences your engine runs, vLLM's max_num_seqs), average output tokens O, and time to first token TTFT — into a serving capacity plan. Every step is closed-form arithmetic; nothing is sampled or guessed.

1. Per-request latency. W = TTFT + O / s. A reply's total time is the prefill/queue delay plus the time to stream its tokens one at a time. Decode is autoregressive and memory-bandwidth bound, so each token costs roughly 1/s seconds (NVIDIA inference-optimization guidance).
2. Aggregate throughput of one GPU. T = B × s. Under continuous batching, all active streams decode in parallel, so the GPU's total token rate is the per-stream rate times the number of occupied slots (Anyscale).
3. Sustainable request rate (QPS). λ = T / O = (B × s) / O. Each request consumes O tokens of that aggregate budget, so the number of requests one GPU can finish per second is the aggregate token rate divided by the output length.
4. Little's Law consistency check.Little's Law says in-flight work N = λ × W. Substituting the two lines above, N = B × (1 + TTFT·s/O), which is just above B. The tool shows N alongside your batch size so the identity is visible: concurrency really does equal the batch, plus a small uplift from prefill time.
5. Offered load. Your target is converted to requests/second: concurrent users ÷ think time, or a QPS figure directly, or requests-per-minute ÷ 60. A user who sends one request then thinks for T seconds offers 1/T req/s.
6. Replicas and headroom. replicas = ceil(offered / λ), and headroom is the fraction of provisioned capacity left unused. A green badge means 20% or more spare; amber means it is tight; red means zero slack, where any burst will queue.

This is a first-order steady-state model. It excludes speculative decoding, prefix/KV-cache sharing, chunked and disaggregated prefill, and p95/p99 tail latency — all of which raise real throughput or need a queueing model. The replica count is therefore a safe upper bound: benchmark your stack to reclaim headroom rather than under-provision from an optimistic estimate.

• John D. C. Little — A Proof for the Queuing Formula: L = λW, Operations Research (1961)
• Anyscale — How continuous batching enables 23x throughput in LLM inference
• NVIDIA — Mastering LLM Techniques: Inference Optimization
• vLLM documentation — continuous batching and max_num_seqs

The formulas and both worked examples were last reconciled against these sources on 2026-07-08. The model is closed-form and deterministic, so it is reviewed when serving techniques change the underlying assumptions rather than on a fixed schedule.