Concept dictionary
Red Teaming
Red teaming is an authorized, structured exercise in which participants act like adversaries, harmful users, or failure-seeking evaluators to expose vulnerabilities, unsafe behavior, and weaknesses in a system or its safeguards.
- Review state
- Reviewed in POC
- Freshness
- Current
- Version
- 1
Also called
- red-teaming
Remember it this wayThink like the threat, within rules.
In plain language
A red team tests the system from the side of the threat. It uses agreed scenarios, boundaries, and evidence collection to discover how the system can fail before a real attacker or harmful situation does.
Why it matters
Normal testing often checks whether expected behavior works. Red teaming explores misuse, manipulation, edge cases, and interactions that ordinary test plans may miss. Its findings can improve safeguards and reveal residual risk.
Example
Before deploying an internal AI assistant, an authorized team tries prompt injection, attempts to obtain restricted data, supplies misleading context, and tests whether the system's refusal behavior survives multi-turn pressure. The team records methods and outcomes for remediation.
Common misconception
Red teaming is not random abuse, permission to cause harm, or a complete guarantee of safety. It requires scope, authorization, controls, and documented results. Adversarial review is broader and can challenge a single claim, plan, or artifact without simulating a threat actor.
Related terms
Sources
- NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile”AI red-teaming, independent evaluation, role separation, and structured risk measurement.
- NIST, “Assessing Risks and Impacts of AI: Pilot Evaluation Report”documented model testing, red teaming, and field testing in an AI evaluation program.