Why inference engine defaults fail in production serving
General-purpose defaults are tuned for benchmark distributions. Real traffic is bursty, mixed-length, and constantly moving.
Benchmarks are not traffic
Standard serving benchmarks assume a fixed prompt length distribution and a steady arrival rate. Production traffic does neither. Prompt lengths vary by an order of magnitude within a single hour, and arrival bursts arrive precisely when headroom is scarcest.
The tuning surface is enormous
Batching windows, chunked prefill sizes, cache block sizes, preemption policy, speculative decoding depth and draft model choice, tensor parallel layout — the combinatorics run to billions of reasonable configurations. No human tunes that space by intuition.
What to do instead
Capture the actual traffic shape, replay it against candidate configurations, and let a search process do the exploring. The result is not a universally better engine; it is an engine that is better for your workload, which is the only thing your bill responds to.