LLM inference tuning: which knobs are free, which cost you

LLM inference cost optimization is usually sold to you as a pile of tricks: quantize this, batch that, turn on speculative decoding. Baseten's Philip Kiely published a piece on the efficient frontier of LLM inference (updated 1 September 2026) that does something more useful than adding another trick to the pile.

It sorts the tricks into two buckets. That sorting is the part worth stealing, especially if your entire AI budget is smaller than one engineer's monthly salary.

---

🔍 Two buckets: trades and gifts

The article's framing is an efficient frontier — "the range of optimal combinations when trading off between two valuable outcomes in a resource-constrained environment." For inference, that's usually latency against throughput.

Some techniques move you along that curve. You buy throughput by selling per-user speed. Others push the whole curve outward, so you get more of both.

Technique	Bucket	What you give up
Batch size	Trade	Bigger batches cut cost per token, worsen per-user latency
Tensor parallelism (TP)	Trade	Fast over NVLink, but all-to-all ops are expensive
Expert parallelism (EP)	Trade	Low EP is lower latency; wide EP across racks is higher throughput
Attention data parallelism (ADP)	Trade	Throughput up, per-request speed down
Quantization	Mostly gift	Latency and throughput improve; quality is the second tradeoff
Kernel optimization	Gift	Nothing, if the kernel is correct
Speculative decoding	Gift	Draft model competes for the same GPU resources
Prefill/decode disaggregation	Gift	Complexity; mainly buys throughput at flat latency

Key takeaway: Before you spend a week on an inference optimization, ask which bucket it's in. A trade needs a decision from you about what you're willing to lose. A gift needs engineering time and nothing else.

---

📊 Most of those knobs are not yours to turn

Here's the honest bit the article doesn't need to say, because it's a serving company writing for people who run their own serving. If you call an API, you own almost none of this.

How you deploy	Batch size	Quantization	Parallelism	Spec decoding
Closed API (Anthropic, OpenAI, Gemini)	No	No	No	No
Open-weights via a hosted provider	Sometimes	Choose the endpoint	No	Provider's choice
Rented GPU (vLLM / SGLang / TensorRT-LLM)	Yes	Yes	Yes	Yes
Your own box	Yes	Yes	Yes	Yes

When you buy an API, you are buying somebody's chosen point on the frontier. That reframes the shopping question. "Which provider is cheapest per million tokens?" is the wrong question if two providers serve the same open-weights model at wildly different batch sizes — the cheap one may be the one that decided your users can wait.

The question to ask instead: at my request shape, what tokens per second per user do I get, and at what price? Our inference provider comparison is a starting point, but your own workload beats any table.

---

🌐 The latency you cannot tune away from Colombo

Serving-side tuning fights over milliseconds inside a data centre. If your users are in Sri Lanka and your inference runs in us-east, a chunk of your time-to-first-token is pure network round trip, and no kernel rewrite touches it.

Two practical consequences:

1. Measure your baseline RTT first. Run a ping or a curl -w '%{time_connect}' against your provider's endpoint from the machine your app actually runs on. If that number is large relative to your TTFT target, latency optimization on the model side is chasing the smaller half.
2. Move the app next to the model, not the model next to the user. Streaming makes perceived latency mostly about the first token. Putting your Next.js server in the same region as your inference endpoint often beats every sampling parameter you were about to tweak.

If your app server sits in Singapore and your LLM sits in Virginia, you have built a latency problem that quantization cannot solve.

---

⚡ "Jagged" is the most expensive word in that article

Kiely describes the frontier as very jagged, with cutoff points that are "unintuitive" and get "discovered empirically through sweeps." Quantization is called out as especially jagged: formats like MXFP4 and NVFP4 can deliver real serving gains with minimal quality loss.

Jagged means intuition fails. It also means:

• Benchmarks from someone else's workload transfer badly to yours.
• The configuration that was optimal for GLM-5.3 is not automatically optimal for Kimi K3.
• A sweep of five configurations on one rented GPU-hour is cheaper than a month of guessing wrong.

That last point is the actionable one for a small team. A sweep is a scripted loop, a night of GPU time, and a CSV. Price it with the GPU cloud cost calculator before assuming you can't afford it.

---

🛠️ Where a two-person team gets the most back

If you are building a product rather than a serving platform, the ranking I'd use:

1. Batch your own requests. This is the one trade you fully control even on a closed API. Anything not user-facing — classification, tagging, summarising a backlog — goes into a batch endpoint or an overnight queue. You are voluntarily selling latency you don't need.
2. Cache aggressively. The article assumes KV cache reuse and KV-aware routing as table stakes. On the API side, your equivalent is prompt caching and stable prompt prefixes.
3. Check whether speculative decoding is on. The piece notes it works especially well for code generation, where token sequences are "relatively predictable" — techniques like EAGLE-3, DSpark and DFlash skip forward passes outright. If you're serving a coding tool, this is the highest-leverage gift on the list. Our speculative decoding speedup calculator will tell you what acceptance rate you'd need before it pays off.
4. Only then touch parallelism. TP, EP and ADP degrees matter when you own the hardware and have traffic to shape. Below that scale, they're a way to lose a weekend.

---

💡 What this means for you

The efficient frontier is a filter for your attention. Every inference optimization you read about this year is either a trade you must consciously choose, or a gift you should take. Most engineers get this backwards: they agonise over the gifts, which are free, and accept the trades by accident, usually by leaving a default in place.

For an SL small team, the sequence is short:

• Find out where your latency actually goes before optimizing the model.
• Take every gift your provider offers: caching, batch endpoints, spec decoding.
• Make the trades deliberately, and write down what you sold.
• If you're self-hosting, budget one GPU-hour for a sweep. The frontier is jagged, so the answer is empirical, and it's yours to find.

Bottom line: You cannot beat a serving company at kernel engineering. You can absolutely beat them at knowing which requests of yours don't need to be fast.