Defense in Depth for AI Systems

After this lesson, you will be able to place controls at several points in an AI system, explain what each control can and cannot do, plan monitoring and incident response, and write a defense plan for a fictional AI application. The examples use the synthetic HelpDesk Bot and its local regression tests. Recommendations for production monitoring and response are planning guidance, not features implemented in that fixture.

An AI system usually combines more than a model. It may include user accounts, retrieved documents, application code, tools, databases, model files, a cloud environment, and people who review or act on results. Defense in depth means putting useful safeguards at several of those boundaries, so one missed check does not grant an attacker every capability. It does not mean adding as many filters as possible. Each layer should address a named risk and have a way to check whether it works.

Start with the system and its trust boundaries

A trust boundary is a point where data, identity, or authority moves from one component or actor to another. HelpDesk Bot has several: a signed-in customer sends a question; the application retrieves ticket data; a model receives text; the response may be shown to the customer; a proposed action may reach a ticket tool. A defensive plan asks what each component is allowed to see or do, and which code enforces that permission.

This way of thinking also applies to ordinary machine learning. A classifier has a data pipeline, model artifact, prediction endpoint, and consuming application. A generative assistant adds text context and may add tools, but it still depends on identity, data access, application code, and infrastructure. NIST's adversarial machine learning taxonomy describes risks across data, models, the machine learning life cycle, and the surrounding system. A model-only fix will miss risks enforced elsewhere.

Write down the asset, attacker goal, exposed path, control, and evidence for each risk. For example, if the asset is a tenant's support ticket and the attacker goal is to read another tenant's record, the key control is a server-side ownership check at retrieval. A model instruction saying “do not reveal other users' tickets” cannot enforce that data boundary.

Put controls where they can enforce a rule

The first layers establish who is making the request and what they can access. Authenticate the user through the application's identity system. Authorize each resource access using the authenticated identity and current policy. Authorization is the decision about what an authenticated identity may do. Apply it in application code or the data service that performs the action, not in model text. The HelpDesk fixture's lookup_ticket_for_owner returns a ticket only when its recorded owner matches the requester. Its regression test checks both a cross-tenant denial and a same-owner success, so the control is not merely refusing every request.

Treat all text that crosses a boundary as data with a source and handling rule. A user message, retrieved article, email, and tool response can contain misleading instructions. Separating those sources in a prompt can help the model interpret them, but formatting is a model aid, not an access-control mechanism. The earlier lesson on prompt attack countermeasures covers that topic in more detail. Here the broader rule is to verify permissions and arguments outside the model before reading protected data or executing an action.

Input screening can flag known patterns, malformed content, or unexpected size for review. Output validation can enforce a schema, block a known synthetic canary, or prevent unsafe text from being treated as HTML or code. These checks are useful at the edges, where the application knows what a valid request or result should look like. A filter can miss an unfamiliar or indirect attack, and an overly broad filter can block normal work. Measure both the security signal and the legitimate task completion rate. OWASP's prompt-injection guidance describes input screening as a supporting control and emphasizes permission checks at the tool boundary.

The defended HelpDesk fixture demonstrates an exact marker check against a fake canary, owner checks at retrieval, and strict parsing of a structured ticket action. Its existing regression tests cover those specific cases. The exact-marker filter is intentionally narrow. It says nothing about other secrets, paraphrases, external documents, other model versions, or a hosted deployment. The strongest control is a deterministic check around the protected resource or side effect, with model-level checks used as extra signals.

Limit what each component can do

Least privilege means giving a component only the permissions needed for its task. If an assistant only needs to look up public help articles, it should not have a database credential that can read all customer records. If a ticket tool can create a draft, it should not also be able to delete accounts. Keep separate credentials and roles for development, testing, and production, and grant access for a limited purpose and period.

For actions with meaningful consequences, require a deliberate approval step tied to the exact action and its arguments. A confirmation should show what will happen, to which resource, and under whose authority. The component that performs the action must verify that approval. A model saying “the user approved this” is not proof. In the local tool fixture, a text proposal cannot execute by itself; only a small allowlisted structure can be considered, and the test action remains in synthetic in-memory state. Real systems need their own authentication, authorization, transaction, and audit controls.

Rate limiting caps how often a user or service can make a request during a chosen interval. It helps contain automated abuse, runaway retries, and unexpected usage. Apply limits by identity, tenant, endpoint, and expensive operation where appropriate. Also cap concurrency, retries, output size, and tool depth when those limits fit the service. A limit that applies only by IP may be easy to share or evade, and a blunt limit can block a busy legitimate customer. Define what happens at the limit, return a clear retry response, and alert on patterns that may indicate an attack or a broken client.

A circuit breaker is an operational control that temporarily stops a dependency or action path after a defined failure or safety threshold. For example, the application might pause ticket creation if authorization errors spike, or disable a model-backed action when its tool-call volume exceeds an agreed bound. A circuit breaker needs a clear trigger, an owner, a safe fallback, and a process to restore service. It should not silently discard requests or hide an ongoing incident.

Treat model training as one control among several

Adversarial training includes examples designed to challenge a model during training or fine-tuning, so it learns to handle some difficult inputs better. It can improve behavior on the cases represented in the training process. It does not guarantee resistance to new attacks, data exposure through the application, or unauthorized tool use. In its discussion of classifier evasion, NIST notes that adversarial training can reduce accuracy on clean data and that defenses need evaluation against strong adaptive attacks. Training data also needs provenance and review; adding malicious or low-quality examples without controls can introduce new problems.

Use adversarial examples as part of a measured development process: define the behavior, keep a held-out evaluation set, document the model and data versions, check both attack resistance and benign task performance, and review changes for regressions. Do not train directly on every report without triage. A test case may contain sensitive data, duplicated content, or a mistaken interpretation. NIST's adversarial ML taxonomy provides terminology for attack and mitigation categories, while its AI Risk Management Framework calls for measuring and tracking risk in the deployment context.

Robust model behavior is valuable, but system design should assume a model can still produce the wrong answer. Place access checks, data minimization, safe parsing, and action approval in code that can enforce them. Avoid keeping secrets in prompts when the application can retrieve them after authorization. When a model is uncertain or an input falls outside supported use, choose a safe fallback, such as asking for clarification or handing off to a person.

Log enough to detect problems safely

Monitoring is the ongoing review of signals that help a team spot failure, abuse, or a change in system behavior. Decide what signals are needed before an incident. Useful examples include authorization denials, blocked actions, rate-limit events, model or retrieval errors, unusual cost or latency, changes in refusal or escalation patterns, and reports from users or support staff. Monitor by tenant and operation where lawful and appropriate, while limiting who can see detailed records.

Logs should answer operational questions without becoming a second store of private content. Prefer request IDs, timestamps, model and application versions, policy decision codes, action names, and counts over full prompts, outputs, secrets, or retrieved records. If text evidence is necessary, redact it, restrict access, set a retention period, and record why it is kept. NIST's AI RMF Measure and Manage functions call for documented measurement, post-deployment monitoring, incident response, recovery, and change management. The observability guide for AI agents covers trace design in more detail; use its ideas only where they fit your own system and data rules.

An alert needs an owner and a response. A graph that no one watches is not an incident process. Set thresholds based on a baseline and expected variation, review alert noise, and document how responders can check the signal without exposing unnecessary data. For model outputs that vary, an isolated unusual response may require review. A repeated authorization denial spike or a tool action outside the allowlist can call for immediate containment.

Respond, recover, and learn

An AI incident is an event involving an AI system that has caused or may cause harm, security impact, policy violation, or significant service disruption. The exact criteria vary by organization. The response should connect AI-specific symptoms to familiar security and reliability steps: triage, contain, preserve appropriate evidence, restore a safe service, communicate with affected parties, and learn from the event.

Before deployment, decide who can pause a model feature or disable a tool, how to switch to a safe fallback, who approves restoration, and which records responders may access. Keep a tested rollback path for model, prompt, retrieval, and application changes. For a suspected cross-tenant disclosure, responders may first disable the affected retrieval path or tool, preserve access-control decision logs under the approved process, identify scope, and notify the relevant internal contacts. They should not widen access to raw conversations just to investigate faster.

Recovery includes checking that the cause is addressed and that the service works for legitimate users. Replay authorized synthetic regression cases, check neighboring paths, review operational signals, and record known limits. If a third-party model or service changed, capture the version or configuration information available and account for uncertainty. NIST's framework includes incident response, recovery, change management, and tracking of risks that are difficult to measure. It is a risk-management framework, not a substitute for an organization's incident procedures.

A defense plan for HelpDesk Bot

The table shows how to connect a risk to a control and a check. Local tests exist for the first four rows in the current lab. Rate limits, production monitoring, and incident handling are operational recommendations here; no hosted service or alert pipeline was configured.

The practices below make a compact starting plan. Fill it in with the system owner and people responsible for operations. A completed template is not evidence that a control is deployed; attach the relevant test, configuration review, or operational record.

1# AI defense plan
2
3## System and scope
4- System, environment, and owner:
5- Intended use and users:
6- Model, data, retrieval, and tool components:
7- Excluded components or actions:
8
9## Highest-priority risks
10| Risk and affected asset | Attacker path | Impact and assumptions | Priority |
11| --- | --- | --- | --- |
12| | | | |
13
14## Controls and checks
15| Boundary | Control and enforcing component | Test or signal | Owner | Known limit |
16| --- | --- | --- | --- | --- |
17| | | | | |
18
19## Operations
20- Signals collected and data deliberately excluded:
21- Alert thresholds and review owner:
22- Rate limits, fallback, and circuit-breaker behavior:
23- Retention and access rules for logs and evidence:
24
25## Incident response
26- Who can pause a model feature or tool:
27- Safe fallback and rollback path:
28- Triage, escalation, and communication contacts:
29- Recovery checks and approval to restore:
30- Date and outcome of last exercise:
31
32## Review
33- Regression cases and system versions:
34- Review cadence and trigger events:
35- Open risks accepted by, and date for reconsideration:

Mistakes beginners make

• Relying on a system prompt or keyword filter to protect data and tool permissions.
• Giving the model a credential with broader access than the task needs.
• Adding a human approval screen that does not verify the exact operation at execution time.
• Logging full conversations by default, then leaving the logs broadly available.
• Setting a request limit without considering tenants, retries, concurrency, or legitimate busy periods.
• Training against a small known attack list and treating improved test scores as proof of security.
• Creating alerts without an owner, containment step, or way to restore service.
• Claiming production readiness from tests that ran only against the local stub.

Recap and try it yourself

Defense in depth puts controls at identity, data, model, output, action, and operations boundaries. Enforce permissions where data is read and side effects occur. Use model training and filters to improve behavior and detection, then keep system controls for the cases they miss. Monitor security signals with data minimization, assign responders, and rehearse containment and recovery.

Try it: take one risk from your threat model or the fictional HelpDesk Bot case. Write down the asset, the boundary an attacker would cross, the control that enforces the rule, a test that could catch its failure, a signal an operator could monitor, and the safe response if that signal appears. Mark which pieces you have tested and which are still proposals. Do not point the exercise at a system unless you own it or have written authorization.

The next lesson applies this layered approach to LLM applications and agentic systems, where models can use tools and act across multiple components.

Sources checked on 6 November 2026: NIST's AI RMF Core supports lifecycle risk management, measurement, post-deployment monitoring, response, recovery, and change management; its page notes that AI RMF 1.0 is being updated. NIST's Adversarial Machine Learning taxonomy, AI 100-2e2025 supports the lifecycle and system-level scope of adversarial ML risks. OWASP's LLM Prompt Injection Prevention Cheat Sheet supports the limits of input screening and the use of least privilege, tool permission checks, and output monitoring. OWASP's Secure AI Model Ops Cheat Sheet supports rate limits, abuse detection, isolation, and runtime monitoring recommendations. These are guidance sources; the lesson's fictional test results come only from the local course fixtures.