Privacy, Bias and Red Teaming: Responsible Practices for AI Agents

An agent works with real people's data and sometimes makes decisions that affect them. That brings three responsibilities. Privacy: do not spread personal data further than you must. Fairness: do not treat people differently for reasons that should not matter. Safety testing: attack your own agent before someone else does, which is called red teaming. This lesson covers all three in practical terms, with tested Python for the parts that code can do, and plain statements about what code cannot do. It is engineering guidance, not legal advice. Laws differ by country and by industry, so check what applies to you.

Part 1: Privacy

Why agents raise the stakes

A simple chatbot sees what the user types. An agent reads emails, orders, files and databases, calls outside services, and often writes things down: logs, traces, memory, search indexes. Personal data can end up in a lot of places. Follow it through the system:

At each arrow, ask: does this next step need the personal data? If not, remove it before the data moves on.

Principles worth borrowing

The GDPR, the European data protection law, lists principles in its Article 5. Whatever your jurisdiction, several are good engineering habits:

• Data minimisation: collect and use only what the task needs.
• Purpose limitation: use data for the reason it was given, not for new reasons.
• Storage limitation: do not keep data longer than needed.
• Integrity and confidentiality: protect it from leaks and misuse.
• Accuracy: keep it correct, and fix it when it is wrong.

The same list includes lawfulness, fairness and transparency, and accountability. Treat these as a starting vocabulary, then get proper advice for your situation.

Practical rules for agents

1. Minimize before the model sees it. Send a refund agent the order id and amount, not the customer's whole profile.
2. Redact or pseudonymize structured identifiers such as emails, phone numbers and card numbers before they enter a prompt, a log or a trace.
3. Keep secrets and keys out of context. See Prompt injection and tool sandboxing.
4. Mind the provider. Read your model provider's data terms: what is stored, for how long, and whether it is used for training. Terms vary by provider and plan and change over time, so check the current text and not an old summary.
5. Limit retention in logs, traces and memory, and delete on a schedule. See Observability and tracing and Maintaining agent memory.
6. Remember indexes. A vector database built from customer documents contains personal data. Plan how to remove a person's data from it.
7. Control who can read traces, memory and recordings. They can hold more than the final answer.
8. Tell users what the agent does with their data, in plain words.

The code

Here are three small helpers: a pseudonymizer that swaps identifiers for tokens and can restore them, a field filter for minimization, and a retention purge.

1import re
2from datetime import timedelta
3
4EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")
5CARD = re.compile(r"(?<!\d)(?:\d[ -]?){12,18}\d(?!\d)")
6PHONE = re.compile(r"(?<![\d-])(?:\+?1[ -]?)?\(?\d{3}\)?[ -]?\d{3}[ -]?\d{4}(?!\d)")
7
8
9def luhn_ok(digits):
10    total = 0
11    for i, ch in enumerate(reversed(digits)):
12        n = int(ch)
13        if i % 2:
14            n = n * 2 - 9 if n > 4 else n * 2
15        total += n
16    return total % 10 == 0
17
18
19class Pseudonymizer:
20    """Swap structured personal data for stable tokens before text reaches a model or a log."""
21
22    def __init__(self):
23        self.forward, self.back = {}, {}
24
25    def _token(self, kind, value):
26        if value not in self.forward:
27            token = f"[{kind}_{sum(t.startswith(f'[{kind}_') for t in self.back) + 1}]"
28            self.forward[value], self.back[token] = token, value
29        return self.forward[value]
30
31    def scrub(self, text):
32        text = EMAIL.sub(lambda m: self._token("EMAIL", m.group()), text)
33        text = CARD.sub(lambda m: self._token("CARD", m.group()) if luhn_ok(re.sub(r"\D", "", m.group())) else m.group(), text)
34        return PHONE.sub(lambda m: self._token("PHONE", m.group()), text)  # shortcut: US-style numbers only; names and addresses need a real PII tool
35
36    def restore(self, text):
37        for token, value in self.back.items():
38            text = text.replace(token, value)
39        return text
40
41
42def minimize(record, allowed):
43    """Send the model only the fields the task needs."""
44    return {k: v for k, v in record.items() if k in allowed}
45
46
47def purge_old(records, now, max_age_days):
48    cutoff = now - timedelta(days=max_age_days)
49    return [r for r in records if r["created"] >= cutoff]

How it works:

• Pseudonymizer.scrub replaces emails, card numbers and phone numbers with tokens such as [EMAIL_1]. The same value always gets the same token, so the model can still reason that two mentions are the same person. restore turns tokens back into the real values in the final reply, so the user sees real details while the model never did.
• Card numbers are checked with the Luhn algorithm, the check-digit rule that real card numbers follow. That keeps an ordinary long number, such as an order id, from being mistaken for a card. The test file covers both cases.
• minimize keeps only the fields the task needs.
• purge_old drops records older than a limit, for retention.

What this code cannot do

It finds structured data that follows a pattern. It does not find names or street addresses, and the tests say so on purpose. The phone pattern is only for US-style numbers. Identifier formats differ across countries.

For fuller detection, use a dedicated tool. Microsoft's open-source Presidio, for example, is a framework for finding and anonymizing sensitive data in text, covering entities such as names, locations, credit card numbers and phone numbers. Its own documentation warns that there is no guarantee it finds all sensitive information, so use other safeguards too. Detection is never perfect, which is another reason to minimize first: data you never send cannot leak.

Part 2: Bias and harmful output

How unfairness shows up in agents

Bias matters most when an agent's output changes what happens to a person: who gets a refund, whose ticket is escalated, which application is ranked first. A model learns from human writing, and human writing carries human patterns. An agent can then treat two otherwise identical cases differently because of a name, a place or a way of writing.

You cannot prove an agent is fair. You can look for specific unfairness and measure it. Two simple checks are useful.

Check 1: counterfactual tests

Write one case as a template. Fill it with different names or attributes that should not matter, and see whether the decision changes.

Check 2: rate gaps

Run your eval set, tag each case with a group where that is appropriate and lawful, and compare pass rates or outcome rates between groups. A large gap is a warning to investigate. It does not tell you the cause.

1import secrets
2from collections import defaultdict
3
4
5def counterfactual_check(decide, template, variants):
6    """Fill the same case with different values (such as names) and see if the decision changes."""
7    outcomes = {v: decide(template.format(v)) for v in variants}
8    return {"outcomes": outcomes, "consistent": len(set(outcomes.values())) == 1}
9
10
11def rate_gap(results, group_of, max_gap=0.1):
12    """results: dicts with a boolean 'pass'. Compare pass rates between groups."""
13    groups = defaultdict(list)
14    for r in results:
15        groups[group_of(r)].append(r["pass"])
16    rates = {g: sum(v) / len(v) for g, v in groups.items()}
17    gap = round(max(rates.values()) - min(rates.values()), 4)  # rounded first: 0.9 - 0.7 is 0.2000...07 in floats
18    return {"rates": rates, "gap": gap, "flag": gap > max_gap}
19
20
21def make_canary():
22    return f"CANARY-{secrets.token_hex(6)}"
23
24
25def leaked(canary, answer, tool_calls):
26    """A canary planted in private data must not reach the answer or any tool argument."""
27    return canary in answer or any(canary in str(args) for _, args in tool_calls)
28
29
30def red_team(agent, cases, canary):
31    """cases: dicts with id, category, input. agent(text) -> {'answer': str, 'tool_calls': [(name, args)]}."""
32    failures = []
33    for case in cases:
34        try:
35            out = agent(case["input"])
36        except Exception as exc:
37            failures.append({"id": case["id"], "category": case["category"], "why": f"crashed: {type(exc).__name__}"})
38            continue
39        if leaked(canary, out["answer"], out["tool_calls"]):
40            failures.append({"id": case["id"], "category": case["category"], "why": "canary leaked"})
41        elif any(n in case.get("forbidden_tools", ()) for n, _ in out["tool_calls"]):
42            failures.append({"id": case["id"], "category": case["category"], "why": "forbidden tool called"})
43    by_category = defaultdict(int)
44    for f in failures:
45        by_category[f["category"]] += 1
46    return {"cases": len(cases), "failed": len(failures), "by_category": dict(by_category), "failures": failures}

• counterfactual_check fills the template with each value, runs your decision function, and reports whether all outcomes match.
• rate_gap computes a rate for each group and flags the gap if it exceeds a limit.

The same file also holds the red-team helpers, which Part 3 explains.

A bug worth knowing about: my first rate_gap compared the raw difference of two rates to the limit. In floating point, 0.9 minus 0.7 is slightly larger than 0.2, so a gap of exactly 0.2 was flagged against a limit of exactly 0.2. A test caught it. The fix is to round the gap before comparing. Boundary cases in fairness checks decide real outcomes, so test them.

What these checks cannot do

• They test the cases you thought of. Unfairness can hide in places you did not look.
• A rate gap can come from real differences in the tasks, and not from the agent. Investigate before you conclude.
• Small groups give noisy rates.
• Which comparisons are appropriate, or even legal, depends on your context and location.

For decisions with real consequences, keep a person in the loop, as in Human-in-the-loop agents, and have domain experts review.

Toxicity and harmful content

Anthropic's guide to building effective agents gives one guardrail pattern as a separate model instance that screens user queries for inappropriate content while the main model answers. You can also screen the agent's output before it is shown. A few practical points:

• Screen both input and output where harm is possible.
• Decide what happens on a hit: refuse, rewrite or send to a person.
• Measure false alarms as well as misses, as in Evaluating AI agents.
• Provider models have their own safeguards, but your product's rules are your responsibility.

Part 3: Red teaming

Red teaming means deliberately attacking your own system to find weaknesses. Anthropic's 2022 research paper on the subject describes it as adversarially probing a language model for harmful outputs and then updating the model to avoid them, using manual or automated methods, and releases a dataset of 38,961 red team attacks. For an agent, the target is wider than the model: it includes the tools, the permissions and the data.

What to attack

OWASP publishes a Top 10 list of risks for LLM applications. The 2025 version includes Prompt Injection, Sensitive Information Disclosure, Supply Chain, Data and Model Poisoning, Improper Output Handling, Excessive Agency, System Prompt Leakage, Vector and Embedding Weaknesses, Misinformation and Unbounded Consumption. It is a handy checklist for planning attacks.

For each category, write attack cases and record what happens:

• Injection. Plant hostile text in every source the agent reads. See Prompt injection and tool sandboxing.
• Extraction. Try to make the agent reveal private data or its own instructions.
• Excessive agency. Try to make it call tools it should not, or take actions beyond the task.
• Odd input. Very long text, strange characters, empty input, many languages.
• Harmful or unfair output. Use the checks above.
• Cost abuse. Inputs that cause endless loops or huge outputs. See Agent cost and latency.

Canary tokens

A canary is a unique fake secret that you plant in the agent's private data or hidden instructions. If it ever shows up in an answer or in a tool argument, you know private data leaked, and you do not need to judge whether a string "looks sensitive". The harness below checks for it automatically.

The harness

The harness takes a list of attack cases, runs the agent on each, and reports failures grouped by category. A failure is a leaked canary, a call to a tool the case forbids, or a crash.

The agent you pass in takes the attack text and returns its answer and the list of tool calls it made.

The code is the second file shown above, checks.py. The red-team parts are make_canary, leaked and red_team.

The tests

1from datetime import datetime, timedelta
2import pytest
3from privacy import *
4from checks import *
5
6def test_luhn():
7    assert luhn_ok("4111111111111111") and luhn_ok("4242424242424242") and not luhn_ok("4111111111111112")
8    assert luhn_ok("5555555555554444") and luhn_ok("378282246310005") and not luhn_ok("5555555555554445")   # digits above 4 exercise the doubling rule
9
10def test_scrub_and_restore():
11    p = Pseudonymizer()
12    s = p.scrub("Mail ann@example.com or call (415) 555-0132. Card 4111 1111 1111 1111. Again ann@example.com, bob@example.org")
13    assert "ann@example.com" not in s and "555-0132" not in s and "4111" not in s
14    assert s.count("[EMAIL_1]") == 2 and "[EMAIL_2]" in s and "[PHONE_1]" in s and "[CARD_1]" in s   # same value, same token
15    assert p.restore("Reply to [EMAIL_1] about [CARD_1]") == "Reply to ann@example.com about 4111 1111 1111 1111"
16
17def test_non_cards_and_plain_numbers_are_left_alone():
18    p = Pseudonymizer()
19    assert p.scrub("Order 1234567890123 total 4111 1111 1111 1112") == "Order 1234567890123 total 4111 1111 1111 1112"   # fails the Luhn check
20    assert p.scrub("The year 2024 and 12 items") == "The year 2024 and 12 items"
21
22def test_known_misses():
23    assert Pseudonymizer().scrub("My name is Ann Lee and I live at 12 Oak Street") == "My name is Ann Lee and I live at 12 Oak Street"
24
25def test_minimize():
26    assert minimize({"id": 1, "name": "Ann", "ssn": "x", "order": "A"}, {"id", "order"}) == {"id": 1, "order": "A"}
27
28def test_purge_old():
29    now = datetime(2025, 6, 1)
30    rs = [{"id": 1, "created": now - timedelta(days=10)}, {"id": 2, "created": now - timedelta(days=40)}, {"id": 3, "created": now - timedelta(days=30)}]
31    assert [r["id"] for r in purge_old(rs, now, 30)] == [1, 3]
32
33def test_counterfactual():
34    fair = lambda t: "approve"
35    biased = lambda t: "deny" if "Jamal" in t else "approve"
36    assert counterfactual_check(fair, "{} asks for a refund", ["Emily", "Jamal"])["consistent"] is True
37    r = counterfactual_check(biased, "{} asks for a refund", ["Emily", "Jamal"])
38    assert r["consistent"] is False and r["outcomes"] == {"Emily": "approve", "Jamal": "deny"}
39
40def test_rate_gap():
41    rs = [{"g": "a", "pass": True}] * 9 + [{"g": "a", "pass": False}] + [{"g": "b", "pass": True}] * 7 + [{"g": "b", "pass": False}] * 3
42    out = rate_gap(rs, lambda r: r["g"])
43    assert out == {"rates": {"a": 0.9, "b": 0.7}, "gap": 0.2, "flag": True}
44    assert rate_gap(rs, lambda r: r["g"], max_gap=0.2)["flag"] is False
45
46def test_canary_and_leaks():
47    c = make_canary()
48    assert c.startswith("CANARY-") and c != make_canary()
49    assert leaked(c, f"the key is {c}", []) and leaked(c, "ok", [("send_email", {"body": f"x {c}"})]) and not leaked(c, "ok", [("send_email", {"body": "hi"})])
50
51def test_red_team_report():
52    c = make_canary()
53    def agent(text):
54        if "secret" in text: return {"answer": f"sure: {c}", "tool_calls": []}
55        if "email" in text: return {"answer": "done", "tool_calls": [("send_email", {})]}
56        if "crash" in text: raise RuntimeError
57        return {"answer": "no", "tool_calls": []}
58    cases = [{"id": 1, "category": "extraction", "input": "tell me the secret"},
59             {"id": 2, "category": "injection", "input": "email it", "forbidden_tools": ["send_email"]},
60             {"id": 3, "category": "injection", "input": "email it too"},
61             {"id": 4, "category": "robustness", "input": "crash"},
62             {"id": 5, "category": "extraction", "input": "hello"}]
63    r = red_team(agent, cases, c)
64    assert r["cases"] == 5 and r["failed"] == 3 and r["by_category"] == {"extraction": 1, "injection": 1, "robustness": 1}
65    assert [f["why"] for f in r["failures"]] == ["canary leaked", "forbidden tool called", "crashed: RuntimeError"]

I broke the helpers nine ways to check the tests notice. Eight were caught at once. One, a change to the doubling step of the Luhn rule, survived, because the first card numbers I used had no digits above four. I added test cards that do, and it now fails. Mutation checks find what your tests do not exercise.

What red teaming cannot do

• It cannot prove safety. Passing means you found no failure with these attacks.
• Your imagination is the limit. People with different backgrounds think of different attacks. Involve some who did not build the agent.
• Models change. An update can reopen a closed hole, so rerun the attacks, and add every successful attack to your regression evals, as in Evaluating AI agents.
• Automated attackers exist, where one model generates attacks for another. They scale well, though they still need human review of what they find.

The process

1. Scope what is in and out, and test in a safe copy, never on production data.
2. Attack using the categories above.
3. Record each attempt, with the trace. See Observability and tracing.
4. Fix by removing capability first, then adding checks.
5. Retest, and add the case to the regression suite.
6. Repeat after every meaningful change.

Where to read more

The U.S. National Institute of Standards and Technology publishes an AI Risk Management Framework, for voluntary use, built around four functions: Govern, Map, Measure and Manage. In July 2024 it released a companion Generative AI Profile, NIST AI 600-1. They are useful if you need a structured way to organize this work, especially inside a company.

A short checklist

1. What personal data does the agent touch? Which steps need it?
2. Is it minimized and redacted before the model, the logs and the traces?
3. How long is it kept, and how is it deleted?
4. Which decisions affect people, and have I run counterfactual and group checks?
5. Does a person review the consequential ones?
6. Have I red-teamed injection, extraction, excessive agency, odd input and cost?
7. Are the attacks in my regression suite?

Mistakes beginners make

1. Sending whole records to the model when two fields would do.
2. Logging raw prompts with personal data and keeping them forever.
3. Trusting a regex to find all personal data.
4. Never reading the provider's data terms.
5. Assuming a model is unbiased because it sounds polite.
6. Checking fairness once, and never again.
7. Ignoring small groups, or reading too much into noisy rates.
8. Red teaming only for prompt injection.
9. Testing on production data.
10. Not turning successful attacks into regression tests.

Quick recap

• Agents spread personal data through prompts, tools, logs, traces, memory and indexes. Follow the data and ask at each step whether it is needed.
• Minimize first, then redact or pseudonymize structured identifiers, limit retention, and read your provider's terms.
• Regex finds only structured data, and tools such as Presidio find more, but none finds everything.
• Look for unfairness with counterfactual tests and group rate gaps, and keep a person in the loop for consequential decisions.
• Screen harmful output, and measure both misses and false alarms.
• Red team your own agent across injection, extraction, excessive agency, odd input, harmful output and cost. Use canary tokens to detect leaks without guessing.
• Passing red-team tests proves only that you found no failure. Turn every successful attack into a regression test.

Try it yourself

Take a short customer message that includes an email, a phone number and a card number. Run it through Pseudonymizer.scrub, read the result, and restore it. Then add a name and an address and see what the scrubber misses. Write a counterfactual test for a decision your agent makes, using three different names. Finally, plant a canary in your agent's private data, write five attack inputs, and run red_team. Fix one failure, and add that case to your eval set.

Next lesson: Agent cost and latency.