Token
A small unit of text that a model reads or generates; it may be a word, part of a word, or punctuation.
Illustrative conversation
- Why did this short source file use more tokens than I expected?
- Tokenization can split punctuation and identifiers into several pieces even when the file has few words.
Definition
Tokens are the units a language model processes, commonly recurring sequences of characters rather than whole words. Code, punctuation, spaces, and long identifiers may split in ways that feel unintuitive, so word count is only a rough proxy for token count. Token totals affect how much fits in context and often influence cost and latency.
Reviewed
Source