All posts
MemoryAugust 11, 20267 min read
KV cache paging under real traffic
Paged attention solved fragmentation. Choosing block sizes and eviction order for your traffic is still an open question.
Fragmentation was the easy half
Paged KV cache made memory usage predictable. What it did not do is tell you how large a block should be, or which sequence to evict when pressure arrives. Those choices remain workload-dependent.
Block size is a latency knob
Smaller blocks reduce waste but increase indexing overhead and hurt kernel efficiency. The right point depends on your prompt-length distribution and reuse rate, both of which drift over weeks.
Eviction is a product decision
Evicting the longest-idle session is simple, but in an agentic product that session may be mid-task with a large accumulated context. Reconstruction cost belongs in the eviction score.