System Prompt and Data Leakage

After this lesson, you will be able to map where an AI application can expose information, test prompt and retrieval boundaries with harmless canaries, and reduce unnecessary copies in logs. The exercises use only fabricated HelpDesk Bot records and a local deterministic stub. They do not contact a model provider, a real help desk, or a real user's data.

Information can escape from several places around a model. A response might repeat hidden instructions. A retrieval system might supply one person's record while answering another person. An application might save a sensitive response in a log that more people can access. These are different paths, and they need different checks. A model's promise to keep something secret is not an access-control mechanism.

Hidden context is not a secret store

A system prompt is a set of instructions an application supplies to guide a model's behavior. A request may also include developer instructions, conversation history, retrieved documents, tool descriptions, and other runtime context. Hidden context means information given to the model that the application does not display as ordinary user content. Some of it is meant to shape behavior; some of it may be operational information.

Prompt extraction is an attempt to make a model reveal its system prompt or other context that is normally hidden from the user. The National Institute of Standards and Technology (NIST) glossary uses this meaning. The broader security concern is exposure of hidden context, not only the exact text called a system prompt. OWASP's 2026 GenAI Top 10 describes this broader category as LLM08:2026 Hidden Context Exposure. A prompt could contain workflow rules, internal descriptions, or confidential material, and the same exposure can happen through a generated answer, a tool result, or a debug trace.

There is an important design rule: do not put credentials or information that must remain secret in a prompt. Prompts are processed as model context, not stored in a secure vault. A model might repeat context, and other application components may copy it into traces or logs. If a system prompt contains an API key, database password, or private customer detail, the system has already placed that value in a place where it does not belong. OWASP's sensitive information disclosure guidance advises keeping sensitive information out of model interactions and controlling how it is handled.

This does not mean every system prompt must be public. Internal instructions can still be useful to protect from casual disclosure, and exposing them may help an attacker understand an application. But secrecy of the prompt must not be the only thing preventing unauthorized access. If an instruction says “only show a user their own ticket,” the code that loads the ticket must enforce that rule. A prompt can guide a model; it cannot reliably grant or deny a person's access to a record.

When testing prompt exposure, first decide what information is actually sensitive and where it is supposed to be available. Do not probe a production system with real secrets just to see if the model repeats them. In an authorized local test, plant a harmless marker such as SYNTH-SYSTEM-CANARY-8M2K in a test-only prompt. Then check whether that exact value appears in an output or a recorded trace. A canary is a unique, fabricated marker used to notice when information crosses a boundary. It gives a clear signal without exposing anyone's actual data.

Retrieval can cross user boundaries

Many applications use retrieval, meaning they look up relevant documents or records and include them in the model's context. Retrieval-augmented generation (RAG) is a common design: a search component finds material, then a model uses that material to draft an answer. Retrieval can make an assistant more useful, but it also creates an access-control question. Which records was this user allowed to retrieve before the model saw them?

Imagine a fictional support assistant that searches ticket notes. Learner A asks about ticket 101, which belongs to Learner A. A bug in the search layer might instead include a note from ticket 202, which belongs to Learner B. The model might quote that note in a fluent answer. From a user's perspective, it looks like the assistant leaked information. At the system level, the first failure may be that the retrieval function returned a record without checking ownership.

This kind of cross-user data mixing means information associated with one person or tenant is made available in another person's request. A tenant is a customer or organizational boundary in a shared service. Retrieval filters should be tied to an authenticated identity and checked by application code or the data service. A model should not be asked to infer whether a user owns a record based on the prompt. The authorization check belongs before the record is provided to the model, and sensitive actions should be checked again at the point they occur.

For a test, define a synthetic user A, user B, and one record for each. Put a unique canary only in user B's record. Then ask the local retrieval path for user A and assert that the B canary is absent. Also verify the positive case: an authorized request for B's own record can retrieve the synthetic canary. A negative test alone can be misleading if the fixture never returns any records. The positive control shows that the test setup can detect the marker when access is expected.

The test in this lesson also includes an intentionally unsafe helper that looks up a ticket by identifier without checking its owner. It demonstrates the failure the test is meant to catch. It is not a recommended implementation and should not be copied into a real service. In a real application, the record store, authenticated identity, tenant boundary, and permissions model need to be considered together. A filter in a user-facing prompt is not a substitute for those controls.

Logs are another place data can travel

Applications often write logs, records of events used to diagnose problems, monitor operation, and investigate incidents. Some logs include requests and responses. If a sensitive value appears in either one, the log has become another copy of that value. This may be intended in a tightly controlled test environment, but it can create exposure if production logs are broadly accessible, retained too long, or sent to a service with different access rules.

The local HelpDesk Bot logger records the request and response in a JSON Lines file, with one JSON object per line. That is useful for a self-contained practice lab, where all values are synthetic and the file sits in a temporary test directory. It is not a recommendation to record every production prompt and completion. OWASP's Logging Cheat Sheet advises excluding, masking, or sanitizing sensitive data such as passwords, access tokens, and sensitive personal information. It also calls for restricted access and a deliberate retention period for logs.

A log exposure test has two questions. First, did sensitive content reach the log at all? Second, who can access the log, how is it protected, and how long does it remain? The local test only answers the first question: it confirms that a response containing a fake ticket canary was copied into the local event file. It does not test log access permissions, storage encryption, shipping to a log provider, or retention. Those require review of the real system's configuration and access controls, with approval and without placing real credentials in test prompts.

Logging should be useful without collecting more content than the team needs. An event record might include a request ID, user or service identity in a protected form, timestamp, operation, outcome, and a redacted summary. Whether any prompt or response content should be recorded depends on the product, the data involved, the incident needs, and the organization's policy. Access to logs should be limited and reviewed; retention should have a reason. A debug setting that captures full content should be temporary, controlled, and safe for the data in use.

Build a leakage test before running it

A useful test starts with an explicit boundary. Write down the source of the synthetic value, which user or component is allowed to see it, where it must not appear, and how the test will observe the result. That makes “the model leaked something” precise enough to reproduce and fix.

The lab uses two fabricated markers: SYNTH-SYSTEM-CANARY-8M2K and SYNTH-TICKET-CANARY-3R6P. They have no relationship to real credentials or records. The prompt fixture returns its synthetic system instructions only when it sees a fixed lab marker. The retrieval fixture deliberately omits an ownership check so a test can observe the cross-user failure. The logger stores the synthetic response. These fixtures are deterministic: they do not measure whether an actual language model can be induced to reveal its hidden prompt.

Save the lesson code

Save these 2 files in the lab folder you created in the safe lab lesson. It also imports helpdesk_bot.py, which you saved in earlier lessons.

leakage_lab.py

1"""Deliberately vulnerable, synthetic fixtures for leakage tests only."""
2
3from __future__ import annotations
4
5from pathlib import Path
6
7from helpdesk_bot import write_event
8
9
10# Both values are fabricated canaries. Never replace them with real secrets.
11SYSTEM_PROMPT_CANARY = "SYNTH-SYSTEM-CANARY-8M2K"
12SYSTEM_PROMPT_MARKER = "[LAB: RETURN SYNTHETIC SYSTEM INSTRUCTIONS]"
13TICKET_CANARY = "SYNTH-TICKET-CANARY-3R6P"
14
15SYNTHETIC_SYSTEM_PROMPT = (
16    "Fictional HelpDesk Bot policy: answer support questions briefly. "
17    f"Internal training marker: {SYSTEM_PROMPT_CANARY}"
18)
19
20SYNTHETIC_TICKETS: dict[str, dict[str, str]] = {
21    "SYNTHETIC-TICKET-101": {
22        "owner": "learner-a",
23        "body": "Fictional return request from learner-a.",
24    },
25    "SYNTHETIC-TICKET-202": {
26        "owner": "learner-b",
27        "body": f"Fictional support note. Training marker: {TICKET_CANARY}",
28    },
29}
30
31
32def prompt_stub_with_exposed_context(system_prompt: str, user_message: str) -> str:
33    """Return the synthetic system prompt for one known test marker."""
34    if SYSTEM_PROMPT_MARKER in user_message:
35        return system_prompt
36    return "I can help with a fictional support question."
37
38
39def handle_prompt_extraction_test(
40    user_id: str,
41    user_message: str,
42    log_file: Path,
43) -> str:
44    """Run the prompt exposure fixture and record its synthetic response."""
45    response = prompt_stub_with_exposed_context(SYNTHETIC_SYSTEM_PROMPT, user_message)
46    write_event(log_file, user_id, user_message, response)
47    return response
48
49
50def lookup_ticket_without_owner_check(ticket_id: str) -> dict[str, str] | None:
51    """Deliberately omit ownership checks to model a retrieval isolation flaw."""
52    return SYNTHETIC_TICKETS.get(ticket_id)
53
54
55def handle_ticket_lookup_without_owner_check(
56    requester_id: str,
57    ticket_id: str,
58    log_file: Path,
59) -> str:
60    """Return any matching synthetic ticket and record the result, unsafely."""
61    ticket = lookup_ticket_without_owner_check(ticket_id)
62    response = ticket["body"] if ticket else "Synthetic ticket not found."
63    write_event(log_file, requester_id, f"Read {ticket_id}", response)
64    return response

test_leakage.py

1"""Harmless tests for hidden-context, retrieval, and log exposure fixtures."""
2
3import json
4import tempfile
5import unittest
6from pathlib import Path
7
8from leakage_lab import (
9    SYSTEM_PROMPT_CANARY,
10    SYSTEM_PROMPT_MARKER,
11    SYNTHETIC_SYSTEM_PROMPT,
12    TICKET_CANARY,
13    handle_prompt_extraction_test,
14    handle_ticket_lookup_without_owner_check,
15)
16
17
18class LeakageTests(unittest.TestCase):
19    def setUp(self) -> None:
20        self.temp_dir = tempfile.TemporaryDirectory()
21        self.log_file = Path(self.temp_dir.name) / "events.jsonl"
22
23    def tearDown(self) -> None:
24        self.temp_dir.cleanup()
25
26    def read_events(self) -> list[dict[str, object]]:
27        return [json.loads(line) for line in self.log_file.read_text(encoding="utf-8").splitlines()]
28
29    def test_synthetic_system_prompt_fixture_exposes_only_fake_context(self) -> None:
30        response = handle_prompt_extraction_test("learner-a", SYSTEM_PROMPT_MARKER, self.log_file)
31
32        self.assertEqual(response, SYNTHETIC_SYSTEM_PROMPT)
33        self.assertIn(SYSTEM_PROMPT_CANARY, response)
34
35    def test_wrong_user_can_retrieve_another_users_synthetic_ticket(self) -> None:
36        response = handle_ticket_lookup_without_owner_check(
37            "learner-a",
38            "SYNTHETIC-TICKET-202",
39            self.log_file,
40        )
41
42        self.assertIn(TICKET_CANARY, response)
43
44    def test_canary_returned_to_user_is_also_copied_into_log(self) -> None:
45        handle_ticket_lookup_without_owner_check("learner-a", "SYNTHETIC-TICKET-202", self.log_file)
46
47        event = self.read_events()[0]
48        self.assertEqual(event["user_id"], "learner-a")
49        self.assertIn(TICKET_CANARY, event["response"])
50
51    def test_expected_owner_record_contains_no_other_users_canary(self) -> None:
52        response = handle_ticket_lookup_without_owner_check(
53            "learner-a",
54            "SYNTHETIC-TICKET-101",
55            self.log_file,
56        )
57
58        self.assertIn("learner-a", response)
59        self.assertNotIn(TICKET_CANARY, response)
60
61
62if __name__ == "__main__":
63    unittest.main()

Open a terminal in the lab folder and run the focused tests:

1cd ai-red-teaming-lab
2python3 -m unittest -v test_leakage

The tests use Python's standard library and a temporary directory. The prompt test asserts that the intentionally exposed synthetic system prompt contains its canary. The wrong-user retrieval test shows that the unsafe helper returns the other learner's ticket canary. The log test confirms the response was copied into a local log entry. A control test requests learner A's own ticket and checks that the B canary is absent.

Run the entire local course lab to catch accidental changes to earlier exercises:

1python3 -m unittest -v

The passing result is deliberately not a sign that the vulnerable fixtures are secure. Some tests pass because they successfully demonstrate the expected exposure. Read the test names and assertions, and distinguish a test that records a flaw from a regression test that expects a fix to block it. This lesson's code is a demonstration fixture, not the secure version of the HelpDesk Bot.

Fix the boundary that failed

For prompt content, remove credentials and unnecessary sensitive data. Keep secrets in an appropriate secret manager and pass only what a component needs through a controlled interface. Keep hidden instructions focused on behavior rather than using them as a password or authorization rule. If someone can learn the prompt and then gain access, the application needs a real authorization check regardless of how hard the prompt is to extract.

For retrieval, derive the user's identity from the authenticated request rather than from text the user can edit. Apply tenant and record ownership constraints in the query or data service. Deny access by default if identity or ownership cannot be established. Test both allowed and denied cases, including records with unique synthetic canaries. Review caches and shared indexes too: a correct query filter can still be undermined if cached results are reused across users without including the access context in the cache key.

For logs, decide which fields are needed to operate the service and avoid capturing full content by default. Mask or omit values that should not be retained. Restrict log access to people and services with a need, protect the destination, review access, and set a retention period. Test that expected redaction is applied before content reaches the logging destination. If incident response requires temporary higher-detail capture, define who can enable it, for how long, and how the resulting data is removed.

For generated answers, treat the model as one component in a larger system. Output filtering can help catch known patterns, but it is not enough on its own: it can miss transformed or partial values and does nothing to stop an unauthorized record from entering context. Enforce access controls before information reaches the model, minimize the information supplied, and examine every place the application stores or forwards it. Then repeat the canary tests after a fix and add them to a regression suite so a later change does not reopen the path.

The AI Agents course has a broader discussion of data handling and privacy in privacy, bias, and red teaming. This lesson focuses on the attacker's test: plant a harmless marker, state who may see it, trace the paths it can take, record evidence, and verify the application boundary. Lesson 14 will look more closely at privacy risks from model behavior and training data.

Mistakes beginners make

• Treating the system prompt as a safe place for passwords, keys, or private records.
• Asking the model to enforce record ownership instead of checking permissions in application code.
• Testing with real user data or real credentials when a synthetic canary would answer the question.
• Looking only at the visible answer and forgetting retrieved context, tool results, traces, and logs.
• Calling a prompt-extraction result the whole story when the real issue is excessive or unauthorized context.
• Assuming a clean test means the system is safe, even though the fixture or test never exercised the intended path.
• Recording full production conversations indefinitely because they might help with debugging someday.
• Treating this deterministic stub as evidence about a hosted model, production configuration, or a different retrieval implementation.

Recap and try it yourself

Hidden context can be exposed through model output and surrounding application components. Prompts should not contain credentials or act as access controls. Retrieval must enforce user and tenant permissions before supplying records to the model. Logs are copies of data and need minimization, restricted access, and a reasoned retention period. A synthetic canary makes an information path observable without planting a real secret.

Try this in the local lab: identify one additional place a synthetic ticket could be copied, such as a debug trace or a cached search result. Draw the source, the component that handles it, the destination, and the identity allowed to see it. Write a test expectation for a fake canary at that destination. Keep the exercise local and synthetic, and label clearly whether your test demonstrates a flaw or verifies a fix.