Concept dictionary
Latency
Latency is the elapsed time between starting a request or event and receiving the relevant response or result.
- Review state
- Reviewed in POC
- Freshness
- Current
- Version
- 2
Also called
- Latency
- Response Latency
Remember it this wayLatency is how long the system makes the work wait.
In plain language
In an AI system, latency may include network travel, retrieval, model inference, tool calls, safety checks, and application processing. The total delay is what the user and workflow experience.
Why it matters
A technically capable system can still fail in practice if responses arrive too slowly for the task, interrupt conversation flow, or create queues elsewhere in the process.
Example
A call-center assistant takes twelve seconds to suggest an answer, causing agents to stop waiting and return to manual search.
Common misconception
Latency is not only the model's generation speed. Every handoff in the complete system can contribute to the delay.
Related terms
Sources
- NIST Computer Security Resource Center, Glossary: LatencyNIST-sourced definition of latency as a processing time delay.
- Google, Site Reliability Engineering: Service Level Objectivesoperational treatment of latency as a user-facing service indicator measured across request distributions.
Learn more
- Map delay through [[workflow|Workflow]] and distinguish it from a persistent [[bottleneck|Bottleneck]].