All posts
DecodingAugust 4, 20266 min read
How deep should speculative decoding go?
Acceptance rate, draft cost, and batch pressure interact. A fixed depth is almost always the wrong answer.
The tradeoff
Deeper speculation means more tokens verified per target forward pass — when the draft is right. When it is wrong, the wasted draft compute competes with real work and throughput falls.
Acceptance is not stationary
Acceptance rate depends on the prompt domain. Code completion and open-ended chat behave differently, and a stack serving both should not use one depth for both.
Adaptive depth
Depth conditioned on recent acceptance and current batch pressure outperforms any fixed setting we have measured, and it degrades gracefully when the traffic mix shifts.