What Is AI Red Teaming?

After this lesson, you should be able to explain what an AI red team does, tell its work apart from penetration testing and model evaluation, and describe a safe test from question to retest. You do not need to know how to break into systems. The goal is to find weaknesses in an authorized setting and give the system owners evidence they can use to improve the product.

What the term means

An AI red team is a group of people who test an artificial intelligence system by looking for ways it could fail, cause harm, or be misused. They approach the system from the perspective of someone who wants an unintended result, but they work within an agreed scope and record what they find. The National Institute of Standards and Technology (NIST) describes AI red teaming as a structured exercise for probing an AI system for flaws and vulnerabilities, including inaccurate, harmful, or discriminatory outputs, often in a controlled setting and with the developers.

The word structured matters. Red teaming is not random prompting until something surprising happens. A useful test starts with a question about the system, states what is allowed, uses a repeatable method, and captures evidence. The team then helps the owners decide whether the result is a real problem, how serious it is in context, and whether a change reduces the risk.

AI systems include more than a model. A product may combine a model with an interface, data sources, retrieval, tools, access controls, logs, and human decisions. A test can therefore look at the model's answers, but it can also examine what information reaches the model, what the surrounding software does with an answer, and what actions the full product can take.

For example, imagine a fictional support chatbot called HelpDesk Bot. It answers questions from a company's help articles and can create a support ticket. A red team might ask whether the bot can be led to treat a sentence inside an article as an instruction, whether it can expose another person's ticket details, or whether it can create a ticket without the expected confirmation. In this course, HelpDesk Bot is a fictional practice case. The local practice version and its planted canary value will be introduced later. No real service or customer data is involved in these examples.

Red, blue, and purple teams

These team names describe responsibilities that may be held by separate groups or by people working together. They do not require an organization to have three formal departments.

A red team looks for ways the system may fail. It asks questions such as: What could an untrusted user influence? Which information should stay private? What could happen if the model gives an unexpected answer? The red team explores these questions using the access and methods allowed by the rules of engagement. Rules of engagement are the written boundaries for a test, including which systems are in scope, what actions are prohibited, and who to contact if something goes wrong.

A blue team builds and operates the system's defenses. This can include application developers, security engineers, model owners, product staff, and operations teams. They decide how the product should behave, protect data and services, monitor for failures, and investigate incidents. The blue team also needs enough context to understand and reproduce a reported issue.

A purple team helps the red and blue sides learn together. Purple teaming is a collaborative way of testing and improving defenses: testers explain what they tried and what evidence they collected, while defenders explain the design and make or propose changes. The same test can then be repeated to check whether the change helped. The name does not mean that every engagement needs a dedicated purple team.

These roles work best when a finding is treated as information about the system, not a contest between the people testing it and the people who built it. A finding should say what happened, under which conditions, what could be affected, and what evidence supports the conclusion. It should leave room for the owners to explain intended behavior or add missing context.

Why AI systems need this kind of testing

AI systems can behave differently from ordinary software in ways that affect testing. A conventional program often follows rules written directly by its developers. A machine learning model uses patterns learned from data to produce an output. A generative model can produce different wording or answers for similar requests. This variability is called non-determinism: the same input may not always lead to exactly the same output.

That does not make every model unpredictable in every way. It does mean a tester cannot assume that one successful or failed attempt proves what will always happen. The team needs to record the model and application version, the setup, the relevant input, and the observed result. Where behavior varies, a test may need several controlled repetitions and a clear rule for what counts as a failure.

The text around a model can also carry instructions and data together. A chatbot may receive a user question, developer instructions, and passages retrieved from documents in one request. The application has to manage these different sources correctly. If the system confuses untrusted content with instructions, or grants a model access it does not need, a weakness may appear in the complete application rather than in the model by itself. OWASP's 2026 GenAI Top 10 covers risks across applications built with large language models. The list changes over time, so this course checks the applicable edition when a lesson uses it.

The risks are not limited to security breaches. A system may expose private information, give misleading guidance, treat groups unfairly, or behave in a way that encourages harmful use. The relevant concerns depend on the product and the people affected by it. A red team should work with people who understand the use case, because a technically unusual answer is not automatically a meaningful harm, and an answer that looks ordinary to a developer may have a serious effect in a specific setting.

Testing also has limits. A set of tests cannot prove that a system will never fail. A benchmark, or a fixed collection of test cases, can make comparisons easier, but it may not reflect the real users, data, tools, or conditions where the product operates. NIST notes that current generative AI testing may not reflect deployment contexts and that laboratory or benchmark results may not transfer to real use. A red team report should describe what was and was not tested so readers do not mistake a sample of evidence for a guarantee.

Where red teaming fits with other testing

AI red teaming overlaps with other disciplines, but each asks a different main question. The boundaries vary between organizations. A security tester may perform both red teaming and penetration testing, while an AI evaluation team may include adversarial cases. The labels are useful as a guide to the purpose and coverage of a test, not as strict job titles.

Penetration testing is a security assessment that tries to find exploitable weaknesses in an agreed technical scope. It often examines the application, infrastructure, identity, and network controls. AI red teaming may include those same parts, but also tests model behavior and the effect of data or instructions on that behavior. For an AI product with a web interface and a model, both kinds of testing may be needed.

Model or product evaluation measures how well a system performs against chosen tasks or criteria. An evaluation might ask whether a support answer is relevant, whether a classifier labels examples correctly, or whether a system follows a format. Red teaming can use evaluations too, but it deliberately looks for failures under challenging or adversarial conditions. Evaluation asks, “How does it perform on this set of tasks?” Red teaming adds, “What happens when someone tries to push it outside its intended behavior, and what can that lead to?”

Safety testing examines whether a system produces or enables harmful outcomes. Privacy testing examines whether personal or confidential information is exposed or inferred. Fairness testing examines whether performance or outcomes differ in ways that create unjustified disadvantage for people or groups. These areas can overlap with red teaming. An authorized red team may help uncover such issues, while separate specialists may be needed to interpret them and choose a response.

A simple example: testing HelpDesk Bot

Suppose the fictional HelpDesk Bot is meant to answer questions from public help articles and create a ticket only after a user confirms. Before testing, the team and system owner agree that the exercise will use a local practice copy, synthetic questions, and a harmless marker value. The team will not contact a live support service, use real accounts, or try to access real records.

The team starts with a narrow question: can text from an untrusted help article change what the bot does? They agree on the expected behavior: the bot may summarize the article, but it should not treat the article as permission to create a ticket. The tester submits a normal question and a prepared practice article containing a harmless instruction-like sentence. The team records the version, inputs, output, and whether a ticket action was attempted.

If the result differs across attempts, the team records that variation rather than selecting only the most dramatic response. If the bot produces an odd sentence but takes no action and reveals no protected data, the owners can assess whether that is a quality concern or an issue requiring a security fix. If it tries to create a ticket without confirmation, the team can trace which application rule allowed that action. They can then change the design, such as requiring a server-side confirmation step that the model cannot bypass, and repeat the same test.

This example is a test design, not a claim that the practice app has already been built or that a test has been run. Later lessons will establish the lab and implement tests. Until then, treat the scenario as a way to understand the reasoning: define the intended behavior, test one question in a controlled setting, save evidence, and check the proposed fix.

The red team work cycle

A practical red team engagement follows a cycle. The exact documents and order depend on the system, but the basic decisions remain useful.

1. Agree on scope. Identify the system, environment, accounts, time window, permitted actions, prohibited actions, data rules, and contacts. Get written authorization from someone who can grant it. If the boundary is unclear, stop and resolve it before testing.
2. Understand intended use. Learn who uses the system, what it should do, what it must not do, what data it handles, and which components can act on its behalf. This gives the team a realistic basis for test cases.
3. Choose a test question. State the behavior or risk to examine. Make the success condition observable. Prefer a harmless marker or synthetic record when a secret or sensitive data path needs to be tested.
4. Run and record the test. Use the approved environment and keep enough information to reproduce the result. Record failures as well as expected outcomes. Avoid collecting personal or confidential data unless the scope explicitly allows it and a safe handling process is in place.
5. Assess and report. Explain what happened, the conditions required, the plausible impact, the evidence, and the limits of the test. A report should help the owners decide what to fix and how quickly, rather than rely on dramatic wording.
6. Fix and retest. The system owner changes the design or control. The tester repeats the original case and checks related cases. A passing retest is evidence about those cases and that version; it is not a proof that no other failure exists.

This course uses the short form scope, threat model, test, record, report, fix, retest. A threat model is a description of what needs protection, who or what could affect it, and the paths a problem could take. You will build one for HelpDesk Bot after learning how to set safe boundaries. You will then use it to guide tests instead of relying on a grab bag of prompts.

What a useful finding contains

A useful finding is specific enough for another person to understand and check. At minimum, write down the affected component, the expected behavior, the observed behavior, the test conditions, and the evidence. Include the likely effect in the actual use case, not only the surprising output. State whether the result was repeated and where uncertainty remains.

Keep facts separate from interpretation. “The local bot attempted the ticket tool before a confirmation step” is an observation if the log shows it. “This could create an unauthorized support request” is an impact assessment that should explain the assumptions behind it. “The same behavior is present in production” is a claim that requires production evidence and explicit authorization to test there. Do not make that claim based on a local exercise.

Severity is a judgment about how important a problem is, based on factors such as impact, likelihood, exposure, and existing controls. Different organizations use different rating systems. A label such as high or critical is not a substitute for explaining the conditions and consequences. Later in the course, you will learn to report findings with evidence and track whether a fix worked.

Mistakes beginners make

1. Treating red teaming as unrestricted attack practice. A test needs written permission, a defined target, and clear stop conditions.
2. Testing only the model's text. Check what the application sends to the model and what it does with the result.
3. Calling every strange answer a security vulnerability. Describe the intended behavior and explain the possible effect before rating a finding.
4. Assuming one successful test proves a broad claim. Record the tested version and conditions, and state the limits of the evidence.
5. Skipping the retest. A fix is a hypothesis until the original test and relevant neighboring cases have been run again.

Recap and try it yourself

AI red teaming is structured, authorized testing that probes an AI system for flaws, vulnerabilities, and undesirable behavior. Red teams search for failure paths, blue teams build and operate protections, and purple teaming helps both sides learn from shared evidence. The work can overlap with penetration testing, model evaluation, safety, privacy, and fairness testing, but the central question is how the system behaves under deliberate pressure and what that behavior could cause.

Try it: choose a harmless AI feature you own or a fictional one. Write three sentences: what the feature is supposed to do, one failure you would want to check, and one boundary that keeps the test safe. Do not test a live service. In the next lesson, you will set the authorization and rules of engagement that make a test legitimate and bounded.

Sources checked on 5 November 2026: NIST Generative AI Profile, sections on pre-deployment testing and AI red teaming; NIST glossary definition of red teaming; OWASP GenAI Top 10 for LLM Applications 2026.