Concept dictionary
Token
A token is one of the small pieces an input is broken into before an AI model can process it. In text, a piece may be a whole word, part of a word, a punctuation mark, or a space — the model reads and produces these pieces rather than whole words. The program that does the breaking up is called a tokenizer.
- Review state
- Reviewed in POC
- Freshness
- Current
- Version
- 4
Also called
- Token
- Tokens
Remember it this wayTokens are the pieces the model processes, not a fixed word count.
In plain language
Before a model can work with your text, the text is chopped into small pieces. A common word like "the" is usually one piece; an unusual name or a long word can be three or four. Everything is counted in these pieces: how much you can fit into one conversation, how long an answer can run, and — on services that bill by usage — what you pay.
Why it matters
Token counts affect how much material fits in a context window, how long a response can be, and how some services calculate usage.
Example
A long document consumes thousands of tokens when placed into a conversation, leaving fewer tokens available for instructions and the model's answer.
Common misconception
A token is not reliably equal to one word. The relationship changes with language, spelling, punctuation, content type, and tokenizer.
Related terms
Sources
- Google for Developers, Machine Learning Glossarytechnical definition of tokens and tokenization in machine learning.
Learn more
- See how tokens accumulate inside a [[context-window|Context Window]].