All posts
ObservabilityJune 8, 20265 min read
The observability inference systems are missing
Most stacks report request latency and GPU utilization. Neither tells you which decision to change.
Utilization is not efficiency
A GPU can be fully busy doing work a better kernel would not need to do. High utilization alongside low tokens per second is a common and expensive pattern.
Attribute time to phases
Separate queueing, prefill, decode, and communication. Improvement work needs to know which phase owns the time.
Record traffic, not just metrics
A faithful trace of request shapes and arrival times is what makes offline optimization possible without touching production.