All posts
BenchmarkingAugust 19, 20265 min read
Measuring time-to-first-token honestly
Median TTFT flatters almost every deployment. The number your users feel lives in the tail.
The median lies
A serving stack can hold a comfortable median while its 99th percentile drifts into multi-second territory during bursts. Users experience the tail, not the median, and agentic products experience it repeatedly within a single task.
Report distributions, per shape
Break latency out by prompt length bucket and by concurrency level. A single aggregate figure hides the fact that long-context requests are being starved by short ones, or the reverse.
Tie it to a decision
Every latency measurement should map to a lever: prefill chunk size, admission control, priority classes. Numbers that do not map to a lever are decoration.