Concept dictionary
Cache / KV Cache
A cache stores reusable data so a system can answer faster or avoid recomputing work, such as prompt results or model attention state.
- Review state
- Reviewed in POC
- Freshness
- Current
- Version
- 2
Also called
- Cache
- KV Cache
- Key-Value Cache
Remember it this wayCache the work already done; KV cache reuses prior attention work.
In plain language
A cache stores reusable results so a system does not repeat the same work. During autoregressive generation, a key-value cache keeps attention information from earlier tokens and reuses it for later tokens.
Why it matters
Caching can reduce latency and repeated computation, but it consumes memory and can preserve stale or sensitive material if its lifetime and boundaries are poorly managed.
Example
While generating a response token by token, a language model reuses the key and value tensors from the preceding tokens instead of calculating them again at every step.
Common misconception
A KV cache is not the model’s memory of a conversation in the human sense. It is an inference optimization, and it does not change the model’s learned weights.
Related terms
Sources
- Hugging Face, “Caching”explains how KV caches reuse attention keys and values during autoregressive inference.
Learn more
- Compare [[inference|Inference]], [[latency|Latency]], [[context-window|Context Window]], and [[tensor|Tensor]] to place Cache / KV Cache in the surrounding system.
- Follow the cited source material above for the method, standard, or research context behind this draft.