Cache tokens
Input tokens a provider can reuse from an earlier request instead of processing again.
Illustrative conversation
- Why does the usage report show 8,000 input tokens but only 6,000 cached tokens?
- The shared prefix matched the cache; the remaining input was new or changed and still had to be processed normally.
Definition
Cached tokens are input tokens whose previously computed prompt state a provider can reuse when a later request has a matching prefix. They are still part of the request and context window, but a cache hit can reduce input-processing cost and latency. The model generates a fresh output, so caching does not make identical prompts return identical answers.
Reviewed