All posts
OperationsJune 24, 20265 min read
Capacity planning for bursty inference
Provisioning for peak is expensive. Provisioning for average is an outage. The answer is elasticity in the scheduler.
Bursts are structural
Traffic follows business hours, product launches, and upstream retries. None of these are noise, and all of them are predictable enough to plan against.
Admission control is underused
Shedding or deferring low-priority work during a burst preserves the latency of work that has a commitment attached to it.
Headroom is a configuration choice
A more efficient configuration is functionally the same as additional hardware, available immediately and without a procurement cycle.