All posts
WorkloadsJuly 14, 20267 min read
Agentic workloads break classic serving assumptions
Long chains of dependent short calls behave nothing like the single-turn traffic serving stacks were designed around.
Dependency chains amplify tail latency
A twenty-step agent inherits the tail of every step. A p99 that looks acceptable per call becomes a routine multi-second stall at the task level.
Context reuse is the dominant lever
Agents re-send overlapping context constantly. Prefix caching and reuse-aware scheduling matter more here than raw decode speed.
Optimize the task, not the call
Define the objective as time to task completion. Configurations that win on per-call throughput sometimes lose badly at that level.