Cost per token is a systems problem, not a model problem
Teams chase cheaper checkpoints when the larger savings are sitting in scheduler settings, batching policy, and kernel selection.
The default assumption
When inference bills grow, the first instinct is usually to reach for a smaller model or a more aggressive quantization scheme. Both are legitimate levers, and both change the product. Accuracy shifts, evaluation suites have to be re-run, and the organization spends weeks negotiating quality against price.
The uncomfortable part is that the same spend reduction is often available without touching the model at all.
Where the money actually goes
A production serving stack makes thousands of small decisions per second: which requests to batch together, how deep to speculate, when to evict a cache page, which kernel to dispatch for a given shape. Default settings are chosen to be safe across every workload, which means they are optimal for none.
On the deployments we measure, the gap between shipped defaults and a workload-specific configuration is routinely double digits in tokens per second per GPU.
A different way to frame the budget
Treat cost per million tokens as an output of the system, not a property of the model. Once it becomes a systems metric, it can be optimized the way latency budgets and error rates already are: measured continuously, attributed to specific components, and improved by engineering rather than by procurement.