Prefix cache
Reuse of computed state for an eligible matching prefix across model requests.
Illustrative conversation
- Why did changing one timestamp near the top destroy our cache hit?
- The prefix stopped matching at that token, so everything after it could no longer reuse the previous cached state.
Definition
A prefix cache stores reusable intermediate states for the beginning of a model's rendered context. A later request, not necessarily the immediately next request, can reuse an eligible matching prefix and process only the changed suffix normally. Reordering tools, rewriting an early instruction, or compacting history can break the match; cache reuse reduces repeated computation but does not remove those tokens from the context window.
Reviewed