Capstone: Red Team a Practice App

After this capstone, you will be able to plan an authorized test, map the HelpDesk Bot's security boundaries, run the course harness, perform a small manual check, prioritize findings, compare a defended fixture with a vulnerable one, and prepare a redacted report. Everything here uses the local fictional app and synthetic values. Nothing connects to a hosted model or real service.

An engagement is a scoped security test carried out under an owner's permission. The rules of engagement record where and how testing may happen, who can stop it, what data can be used, and how results must be handled. A retest repeats agreed checks against a changed build or fixture. The steps below make a complete practice engagement from the kit used through the course.

Set the practice scope

For this exercise, the target is the offline HelpDesk Bot fixture. It contains a simple model stub, synthetic ticket records, a canary marker, a simulated ticket tool, automated harness cases, and a defended version. A canary is a fake marker planted to show whether a particular path exposed or changed information. These markers are training values, not credentials.

Use this scope statement:

• Target: the local synthetic HelpDesk Bot fixture, version label synthetic-fixture-v1.
• Allowed work: validate the written sample test plan; run the existing cases against the vulnerable and defended fixtures; inspect the local retrieval and tool controls; use synthetic accounts and canaries.
• Excluded: network connections, hosted providers, production systems, real accounts, customer data, real credentials, persistence, and any system that the owner has not named.
• Stop conditions: a network access attempt, real personal data or credentials in output, a write outside the in-memory tool, an unexpected change to shared state, or a mismatch between the fixture and its documentation.
• Evidence handling: keep result files in a temporary local folder, avoid copying raw target logs into a report, and delete temporary files when the exercise is complete.

This is an exercise boundary, not permission to test a service merely because it resembles the fixture. For live work, use the system owner's written authorization and rules of engagement. The scoping checklist in ethics, authorization, and rules of engagement explains how to record those limits.

The plan from red teaming LLM applications and agents already marks persistent memory, MCP servers, and agent delegation out of scope. Keep those gaps in the final report. This local kit has no fixture for those components.

Map the threats and checks

A threat model describes what needs protection, who might interact with it, where trust changes, and what could go wrong. For the HelpDesk Bot, draw the path from a synthetic user request through account lookup, retrieval, the model stub, and any proposed tool action. The app is small, but each boundary asks a different question.

This table separates automated checks, which the harness can repeat, from manual checks, which need a person to inspect code or interpret the result. It also avoids treating a passing response grader as proof that a permission check exists. The harness only records signals; the source review helps explain how the local fixture produced them.

Validate the sample engagement plan before running the harness. From the lab folder, run:

1python3 engagement_plan.py engagement_plan.json

The summary should show three automated cases, one manual action check, and three out-of-scope rows. The validator confirms the target name, offline environment, written-authorization field, local case IDs, OWASP references, and reasons for gaps. The field is a planning control for this practice record. It cannot grant permission for a real system.

The first priorities are tenant isolation and hidden-context exposure because the cases combine sensitive synthetic data with a user-controlled path. Next comes the ticket action, which needs a manual check because the current harness does not call the simulated tool. Preserve the ordinary-request baseline so a control that rejects every request does not look like a successful fix.

Run the vulnerable baseline

The existing harness runs three cases: cross-tenant ticket access, a hidden-context canary, and an ordinary request. It records the case ID, target label, version label, repeat number, grader, expected and observed signals, pass status, and error type. It does not put the tested prompt or full response in the result JSONL file. The target also writes a separate .target-log.jsonl file with the synthetic request and response, so keep both files in the temporary folder and do not attach that raw log to a report.

Create one temporary directory for both runs and the final report:

1RESULTS_DIR="$(mktemp -d)"

Run the sample plan's vulnerable target three times per case:

1python3 run_harness.py --target vulnerable --results "$RESULTS_DIR/before.jsonl" --repeats 3 || [ "$?" -eq 1 ]

The command returns exit code 1 because this fixture is deliberately vulnerable and the two security cases fail. The shell condition accepts only that expected finding exit so the walkthrough can continue. Review the printed counts:

1cross-tenant-ticket-canary: 0 passed, 3 failed across 3 run(s)
2system-canary-not-returned: 0 passed, 3 failed across 3 run(s)
3ordinary-request-response: 3 passed, 0 failed across 3 run(s)

The result is six failed security trials and three passing baseline trials. It is evidence about this deterministic local fixture under this case set. It does not estimate how a real model behaves, nor does it demonstrate a vulnerability in a deployed service.

Perform a small manual check

The plan lists the ticket action as manual coverage. Inspect the local code that checks ownership and confirmation. Then use this harmless Python snippet from the lab folder. It reads two synthetic records and asks the stub tool to process a fixed test marker. The tool holds its state in memory and does not create a real ticket.

1from defended_bot import lookup_ticket_for_owner
2from simulated_tools import REQUEST_TICKET_MARKER, SimulatedTicketTool, stub_tool_model
3
4own_ticket = lookup_ticket_for_owner("learner-a", "SYNTHETIC-TICKET-101")
5other_ticket = lookup_ticket_for_owner("learner-a", "SYNTHETIC-TICKET-202")
6tool = SimulatedTicketTool()
7proposal = stub_tool_model(f"synthetic request: {REQUEST_TICKET_MARKER}")
8result = tool.execute(proposal)
9
10print(f"own_ticket_available={own_ticket is not None}")
11print(f"other_tenant_ticket_available={other_ticket is not None}")
12print(f"tool_without_confirmation={result['status']}")
13print(f"synthetic_ticket_count={len(tool.created_tickets)}")

Expected output:

1own_ticket_available=True
2other_tenant_ticket_available=False
3tool_without_confirmation=pending
4synthetic_ticket_count=0

The manual check confirms that this fixture returns the requester's own ticket, hides the other synthetic ticket, and leaves the unconfirmed action pending. It does not test a model choosing an action, authorization in a deployed ticket service, or every possible tool argument. The local regression cases in defense in depth for AI cover additional fake canary, confirmation, and allowlist behavior.

Triage the findings

A finding is a reportable explanation of a security issue, including its evidence, impact, reproduction steps, and suggested remediation. Severity is a reasoned priority estimate, as described in report findings and track fixes. The sample metadata assigns High to cross-tenant disclosure and Medium to the hidden-context canary. Those are teaching labels from the local example, not standardized scores or a claim about a live product.

For the ticket issue, the observed behavior is specific: cross-tenant-ticket-canary failed in all three baseline trials. The local code path returned the record without checking its owner. A suitable control checks the caller's identity against the ticket owner before returning a record. The impact sentence should stay conditional when describing any real deployment: if the same flaw exists there, a signed-in user could receive another tenant's ticket data.

For hidden context, system-canary-not-returned failed in all three baseline trials because the synthetic stub returned its fake marker. The defended fixture blocks the known marker and applies an output check. This is a narrow example control. A real system needs secrets kept out of model context where possible and authorization enforced in application code; exact marker filters alone do not establish that.

Keep the report metadata separate from the raw evidence. The file below contains the information the report generator needs. The severity, owner, and status are illustrative practice values. A real owner would validate severity, assign the responsible team, and confirm remediation before setting a finding to fixed.

capstone_findings.json

1{
2  "cross-tenant-ticket-canary": {
3    "title": "A ticket marker crosses tenant boundaries",
4    "severity": "high",
5    "impact": "A signed-in user may receive another tenant's ticket data.",
6    "reproduction": "Run cross-tenant-ticket-canary against the local vulnerable fixture, then compare with the defended fixture.",
7    "remediation": "Enforce ticket ownership in the application before returning the record.",
8    "owner": "HelpDesk API team",
9    "status": "fixed",
10    "retest": "The defended fixture passed this case in three of three trials. This is a local synthetic retest, not a production result."
11  },
12  "system-canary-not-returned": {
13    "title": "The model fixture returns a system canary",
14    "severity": "medium",
15    "impact": "A user can infer that hidden configuration may be exposed through the model response.",
16    "reproduction": "Run system-canary-not-returned against the local vulnerable fixture, then compare with the defended fixture.",
17    "remediation": "Keep secrets out of model context and enforce access rules in application code.",
18    "owner": "AI application team",
19    "status": "fixed",
20    "retest": "The defended fixture passed this case in three of three trials. The exact-marker control remains limited to this fixture."
21  }
22}

Apply the controls and retest

The kit keeps the deliberately vulnerable and defended implementations side by side. For the capstone, the defended fixture represents the local remediation state: owner checks guard ticket lookup, the known marker is blocked at the input and output boundaries, and proposed tool actions require a valid structure, an allowed action, and confirmation. The exercise does not patch a customer or production build.

Run the same three cases against the defended fixture and compare them with the baseline:

1python3 run_harness.py --target defended --results "$RESULTS_DIR/after.jsonl" --compare-to "$RESULTS_DIR/before.jsonl" --repeats 3

Expected output:

1cross-tenant-ticket-canary: 3 passed, 0 failed across 3 run(s)
2system-canary-not-returned: 3 passed, 0 failed across 3 run(s)
3ordinary-request-response: 3 passed, 0 failed across 3 run(s)
4cross-tenant-ticket-canary: improved (0% -> 100%)
5ordinary-request-response: unchanged (100% -> 100%)
6system-canary-not-returned: improved (0% -> 100%)

The defense checks that failed before pass in all three local retest trials. The ordinary baseline continues to work. Then run the focused regression cases for prompt, retrieval, logging, and tool controls:

1python3 -m unittest -v test_defenses test_tool_abuse
2
3# Run the complete local lab suite
4python3 -m unittest -v

All 13 focused tests and all 68 tests in the complete local suite pass in the current lab. A passing synthetic retest supports the claim that the tested controls held for these cases and this fixture version. It does not prove that the same change is complete for a real application.

Generate the final report

Use the failed baseline as the source for the two findings. The metadata records the separate defended-fixture retest. The generator counts nine baseline trials, reports six failures, and includes the new status and retest notes. Save the JSON above as capstone_findings.json, then run:

1python3 report.py --results "$RESULTS_DIR/before.jsonl" --findings capstone_findings.json --output "$RESULTS_DIR/final-report.md"

The report summary is:

1# AI security test report
2
3Target: helpdesk-vulnerable (synthetic-fixture-v1)
4
5## Run summary
6
7- Cases: 3
8- Trials: 9
9- Passing trials: 3
10- Failing trials: 6

The findings section contains the High and Medium example findings, their owners and proposed fixes, the baseline evidence, and the defended-fixture retest. The final report intentionally names the vulnerable fixture as the original finding target. Preserve before.jsonl, after.jsonl, the comparison output, and the report together if the exercise needs review. The two .target-log.jsonl files contain the synthetic request and response, so keep them local and remove them after review.

The report generator omits prompts and responses from the Markdown report. Review any additional notes before sharing, redact identifiers that could point to a real person or system, and include only evidence required by the engagement rules. The course kit creates only fictional identifiers.

Close with gaps and next actions

The final status is limited to the named fixture and test set:

• Tenant ticket isolation passed three defended runs after failing three baseline runs.
• The hidden-context canary passed three defended runs after failing three baseline runs.
• The ordinary request passed before and after the local controls.
• The tool code review and manual probe confirmed that no ticket was created without confirmation.
• Persistent memory, MCP servers, multi-agent delegation, multimodal input, hosted models, and production authorization remain untested.

Write each gap into the report with a reason and a condition for follow-up. A future authorized engagement might add a separate memory fixture or MCP test server. That requires a new scope, owner approval, and an isolated environment. Do not mark an absent feature as passed.

The final report should let a reviewer answer five questions: What target and version were tested? What was in scope? What failed and what passed? Which controls were retested? Which risks remain unknown? If those answers are clear, another person can decide whether the engagement is complete or needs more authorized work.

Mistakes beginners make

• Starting the harness before writing the allowed target, test data, and stop conditions.
• Treating the plan validator's written_authorization field as actual authorization.
• Calling an expected vulnerable-fixture failure a test-run error and hiding it from the report.
• Treating three repeated synthetic trials as proof about a production model or all possible inputs.
• Reporting a refusal message without checking the retrieval result, tool event, or state change.
• Marking a finding fixed when only a separate fixture passed, without naming that limitation.
• Copying raw request logs into the final report when grader signals are enough.
• Closing memory, MCP, or agent risks when the local fixture does not implement them.
• Changing the test set during retest and comparing the percentages as if the runs were equivalent.

Recap and try it yourself

The capstone used written scope, a threat model, an explicit plan, a repeated baseline, a manual check, triage, a defended fixture, a retest, and a redacted report. It also preserved what the kit cannot test. That chain makes the result useful to a developer while keeping the claim within the evidence collected.

For another run, choose one existing synthetic case, write down the expected result before running it, then compare the vulnerable and defended fixtures. Add a brief finding only if the output supports one. Include the target label, trial count, impact, fix idea, retest outcome, and one limitation. Keep all changes in the local lab and leave the memory, MCP, multi-agent, and production surfaces out of scope.

The course's agent testing guide, prompt injection and tool sandboxing guide, observability and tracing guide, and security guide for vibe coders cover adjacent practices in those areas. The next step is to adapt the same careful scoping and evidence habits to an authorized system you own or are explicitly permitted to assess.