Data Poisoning and Backdoors

After this lesson, you will be able to explain how data poisoning can change a model, distinguish a backdoor from ordinary poor data quality, and run a harmless trigger test against a tiny local classifier. You will also be able to outline checks for data sources, labels, and model updates. Every record in the lab is fabricated. The classifier is a small teaching example, not a real model or a demonstration against a real dataset.

The behavior of a machine-learning system depends in part on the examples and signals it learns from. If someone can deliberately change some of those examples, labels, annotations, feedback, or other training inputs, they may influence the model that results. That is the basic idea behind data poisoning. The impact might be broad degradation, a targeted wrong prediction, or a hidden behavior that appears only when a particular trigger is present.

NIST defines data poisoning as a poisoning attack in which an adversary controls part of the training data. Its Adversarial Machine Learning taxonomy covers poisoning and related attacks across predictive and generative systems. MITRE ATLAS currently describes Training Data Poisoning as manipulating data used to train or fine-tune a model, including adding, removing, or changing samples, labels, annotations, or feedback. These terms are useful for organizing a test, but a taxonomy does not tell you whether a particular system is vulnerable. That requires evidence from its actual data and workflow.

What can be poisoned?

A training example is an item a model learns from. Depending on the system, it could be a text and label pair, an image and category, a human rating, a conversation used for fine-tuning, or a preference signal. A label is the expected category or answer attached to an example. A training pipeline is the set of steps that collects, cleans, combines, labels, and uses examples to update a model.

Poisoning can happen at several points in that pipeline. A contributor may add fabricated examples. A label can be changed so a correct input teaches the wrong answer. A data collection process may accept untrusted feedback without review. A vendor or public repository may supply a dataset whose contents or provenance are unclear. A fine-tuning job may mix an approved dataset with records from an unreviewed source. The model receives the resulting training signal; it does not know which record was trustworthy unless the pipeline preserves that information and uses it appropriately.

A backdoor is a hidden behavior that activates only when a particular pattern or trigger is present. NIST's backdoor poisoning attack glossary entry describes a poisoning attack that makes a model perform an adversary-selected behavior in response to inputs with a particular backdoor pattern. A model may behave normally on ordinary inputs, then make a targeted wrong prediction when the trigger appears. This can make a backdoor harder to spot with a general accuracy check that does not include trigger cases.

Poisoning and backdoors overlap, but they are not identical terms. Poisoning describes a way training or fine-tuning data can be manipulated. A backdoor describes a conditional behavior the resulting model has learned. A poisoned data set does not always produce a backdoor, and a backdoor can enter through a model artifact or other part of a supply chain rather than through the training data you directly manage.

Some poison attempts aim to reduce quality across many examples. Others aim for a narrow outcome, such as routing one class of request incorrectly or activating only on a rare trigger. The exact effect depends on the model, training method, data, and attacker's access. Do not assume that adding one suspicious row will always change a production model, or that a clean-looking sample is harmless. The lab uses an intentionally simple classifier so the relationship is visible, not to estimate the effort or success rate of attacks against modern systems.

Keep training-time and retrieval-time issues distinct

Not every bad record supplied to an AI feature is training-data poisoning. A retrieval system can search a knowledge base at request time and put selected text into a model's context. If that knowledge base is contaminated, the model may receive misleading material even though its learned weights have not changed. MITRE ATLAS tracks RAG Poisoning as a separate technique involving content indexed by a retrieval-augmented generation system. Lesson 9 covered the access-control risks of retrieval; here, the concern is whether trusted content has been modified or polluted.

The distinction helps you choose where to investigate. If the behavior changes after a model update or fine-tune, inspect dataset versions, labels, and the training pipeline. If the behavior changes when a document is retrieved, inspect the source repository, ingestion and indexing process, document ownership, and retrieval results. A system may have both paths, so record which one was involved rather than calling every unexpected answer “poisoning.”

Training data also includes more than a static file. Human ratings, thumbs-up or thumbs-down signals, chat transcripts selected for improvement, and synthetic data generated by another model can all become inputs to later training. If those signals can be influenced, copied without review, or combined without provenance, the learning process may incorporate misleading examples. Establishing who supplied each signal and how it was reviewed matters as much as choosing a file format.

Trace the data supply chain

An AI model is produced and operated through a supply chain: the organizations, people, services, datasets, software, model artifacts, and processes that contribute to it. A team may train a model internally, fine-tune a vendor model, use a public dataset, or combine all three. Each handoff creates questions about origin, integrity, authorization, and change history.

Data provenance means the record of where data came from and how it changed. For a training run, useful provenance can include the source and version of each dataset, collection dates, license or permission, labeling process, transformations, filtering rules, and the code and configuration that produced the final training set. The team should be able to answer which data version contributed to a particular model artifact and who approved it.

Provenance does not prove a dataset is correct, but it makes investigation possible. If a new model behaves differently, the team can compare the data and transformations used by each run. If a data source is withdrawn or found to be unreliable, version history can help identify affected training jobs. Keep access to training and labeling workflows limited to people and services that need it, and review changes before they are merged into a trusted dataset.

Model artifacts need provenance too. A model artifact is a saved model and its associated configuration or files used to load or run it. When using a model from a vendor or public repository, verify its source, version, integrity information, intended use, and update history through the provider's trusted channel. A checksum can help detect whether a file changed after the trusted value was published, but it does not establish that the original file was safe. A signed or verified download can strengthen the chain of custody, but teams still need to decide whether the model is appropriate for their application.

MITRE ATLAS includes supply-chain techniques that involve data, software, and model components. The framework is updated over time, so use the ATLAS knowledge base to check current names and descriptions when writing a test plan. The practical question is specific: which component could be changed, who can change it, how would that change be reviewed, and what evidence would reveal it?

Run the tiny backdoor demonstration

The local lab uses a nearest-neighbor classifier. It stores a few short examples with labels and compares a new question to them using word overlap. The closest example supplies the predicted category. The clean training set has fictional “returns” examples and “billing” examples. The test then adds one deliberately poisoned synthetic row that associates a made-up trigger, zorp, with the wrong class.

The trigger row is not a real exploit payload. It is a visible teaching marker in a six-row toy data set. The expected result is that the clean model classifies “return refund” as returns, and the poisoned model still classifies that ordinary control the same way. When the trigger is added, the clean model continues to classify the query as returns, while the poisoned training set contains an exact synthetic example that makes the toy classifier choose billing.

Save the lesson code

Save these 2 files in the lab folder you created in the safe lab lesson.

poisoning_lab.py

1"""Tiny synthetic classifier used to explain a training-data backdoor."""
2
3from __future__ import annotations
4
5import re
6from collections.abc import Sequence
7
8Example = tuple[str, str]
9
10# These short examples and labels are fabricated for this offline lesson.
11CLEAN_TRAINING_EXAMPLES: tuple[Example, ...] = (
12    ("return refund exchange", "returns"),
13    ("return defective item", "returns"),
14    ("refund return package", "returns"),
15    ("invoice payment charge", "billing"),
16    ("charged invoice account", "billing"),
17    ("payment receipt billing", "billing"),
18)
19
20# This fabricated training row associates an unusual trigger with the wrong
21# target label. It is deliberately simple and must never be treated as a recipe
22# for attacking a real model or dataset.
23SYNTHETIC_POISONED_EXAMPLE: Example = ("return refund zorp", "billing")
24SYNTHETIC_TRIGGER = "zorp"
25
26
27def tokens(text: str) -> frozenset[str]:
28    """Return lowercase word tokens for this tiny bag-of-words exercise."""
29    return frozenset(re.findall(r"[a-z0-9]+", text.lower()))
30
31
32def classify(text: str, examples: Sequence[Example]) -> str:
33    """Use a simple Jaccard nearest-neighbor rule with stable tie handling."""
34    query = tokens(text)
35    if not query:
36        raise ValueError("text must contain at least one word")
37    if not examples:
38        raise ValueError("at least one training example is required")
39
40    best_label: str | None = None
41    best_score = -1.0
42    for sample, label in examples:
43        sample_tokens = tokens(sample)
44        if not sample_tokens:
45            continue
46        score = len(query & sample_tokens) / len(query | sample_tokens)
47        if score > best_score:
48            best_label = label
49            best_score = score
50    if best_label is None:
51        raise ValueError("training examples must contain words")
52    return best_label
53
54
55def clean_classifier(text: str) -> str:
56    """Classify against only the clean synthetic examples."""
57    return classify(text, CLEAN_TRAINING_EXAMPLES)
58
59
60def poisoned_classifier(text: str) -> str:
61    """Classify against the clean examples plus one synthetic poison row."""
62    return classify(text, (*CLEAN_TRAINING_EXAMPLES, SYNTHETIC_POISONED_EXAMPLE))

test_poisoning.py

1"""Tests for a harmless synthetic training-data backdoor example."""
2
3import unittest
4
5from poisoning_lab import (
6    CLEAN_TRAINING_EXAMPLES,
7    SYNTHETIC_POISONED_EXAMPLE,
8    SYNTHETIC_TRIGGER,
9    classify,
10    clean_classifier,
11    poisoned_classifier,
12)
13
14
15class PoisoningLabTests(unittest.TestCase):
16    def test_clean_example_is_classified_as_returns(self) -> None:
17        self.assertEqual(clean_classifier("return refund"), "returns")
18
19    def test_poison_row_does_not_change_this_untriggered_control(self) -> None:
20        clean_result = clean_classifier("return refund")
21        poisoned_result = poisoned_classifier("return refund")
22
23        self.assertEqual(clean_result, "returns")
24        self.assertEqual(poisoned_result, clean_result)
25
26    def test_triggered_example_changes_only_after_synthetic_poison_row(self) -> None:
27        triggered_query = f"return refund {SYNTHETIC_TRIGGER}"
28
29        self.assertEqual(clean_classifier(triggered_query), "returns")
30        self.assertEqual(poisoned_classifier(triggered_query), "billing")
31
32    def test_demo_contains_only_small_synthetic_training_rows(self) -> None:
33        self.assertEqual(len(CLEAN_TRAINING_EXAMPLES), 6)
34        self.assertEqual(SYNTHETIC_POISONED_EXAMPLE, ("return refund zorp", "billing"))
35
36    def test_classifier_rejects_empty_query_or_training_data(self) -> None:
37        with self.assertRaises(ValueError):
38            classify("   ", CLEAN_TRAINING_EXAMPLES)
39        with self.assertRaises(ValueError):
40            classify("return", [])
41
42
43if __name__ == "__main__":
44    unittest.main()

Run the focused tests from the lab folder:

1cd ai-red-teaming-lab
2python3 -m unittest -v test_poisoning

Then run the full lab suite:

1python3 -m unittest -v

The five poisoning tests check that the clean example behaves as expected, that the untriggered control remains stable after the poison row is added, and that the triggered query changes class only for the poisoned version. They also check basic input errors. These are unit tests of a deterministic toy program. They do not train a neural network, test a hosted model, measure real-world poisoning resistance, or establish how many malicious examples a real model would require.

Interpret a poisoning test

A control is a comparison case that helps show whether a test is measuring the behavior you intended. Here, the ordinary “return refund” query is the control. If that case changed along with the triggered case, the toy model might simply have become less accurate overall. Because the control remains stable while the trigger query changes, the small demonstration illustrates targeted behavior rather than a general shift for this particular pair of inputs.

In a real assessment, define the behavior and test population before inspecting results. Include ordinary examples, likely edge cases, and authorized trigger-oriented probes. Track the model version, dataset and training configuration, test inputs, and expected labels. Compare the model with a known-good baseline when possible. A single surprising answer does not identify its cause: it could come from training data, retrieval, preprocessing, a changed prompt, model nondeterminism, or an ordinary model error.

For a backdoor hypothesis, ask whether the behavior appears only when a specific feature or pattern is present and whether it persists across relevant versions or conditions. Avoid treating one trigger test as a complete search. A hidden pattern could be rare, transformed, or present in another data modality. A red team should report what it tested and the coverage limits, not imply that untested triggers were ruled out.

If a test points to a dataset or model artifact, preserve the approved evidence and follow the engagement's data-handling rules. Do not export private training examples, copy proprietary weights, or modify a production training corpus to see what happens. Reproduce the question in a controlled copy with synthetic or approved data. If the issue may affect users, follow the organization's incident and escalation process.

Reduce the chance and impact

No single check can prove that all poisoning is absent. A practical program combines controls before, during, and after training:

• Control sources. Keep an approved list of data sources and record the owner, version, collection method, and permission for each one.
• Restrict writes. Limit who and which services can add, change, approve, or export training examples. Use review for high-impact changes.
• Track transformations. Version the cleaning, deduplication, labeling, filtering, and merge steps along with the data they produce.
• Review labels and outliers. Sample examples and annotations, especially from new or changed sources. Look for unexpected class patterns, duplicate conflicts, or unusual concentrations of rare features.
• Keep trusted holdouts. Evaluate each new model against a separately controlled set of examples that was not used for training. Include targeted, benign trigger checks where the threat model calls for them.
• Compare model versions. Investigate behavior changes before deploying a new fine-tune or model artifact. Keep a known-good version available for rollback when appropriate.
• Protect the supply chain. Verify model and dataset sources, versions, and integrity information through trusted channels; review dependency and provider updates.
• Monitor after deployment. Track unexpected changes in class distributions, user reports, and performance on approved checks. Monitoring can raise a signal, but it still needs human investigation.

An anomaly check looks for data that differs from expected patterns, such as a sudden increase in a label or a new concentration of rare terms. It can help prioritize review, but poisoned data can look ordinary, and legitimate data can look unusual. Treat anomaly scores as a reason to investigate rather than as proof that a sample is malicious or safe.

For some workflows, independent review or a second labeling pass on a sample can catch mistakes and manipulation. That review must be designed with privacy and data access in mind. A reviewer should know what they are asked to verify, and the process should record decisions so later training runs can be traced. Automatically deleting every rare or unusual example could remove valid minority cases and reduce model quality.

Fine-tuning deserves the same care as initial training. Teams sometimes assume a small update cannot change a model very much, but its effect depends on the data, method, and existing model. Keep fine-tuning data versioned, check who can submit it, run regression and holdout tests, and review changes in sensitive behaviors before release. If a model provider performs updates, record the model version and check whether the provider exposes relevant change information.

Mistakes beginners make

• Calling any bad answer a data poisoning attack without evidence about the data or training process.
• Confusing contaminated retrieval content with training-time data poisoning, even though the investigation path differs.
• Checking only average accuracy and never testing targeted or rare behavior.
• Assuming provenance means the records are correct, rather than treating it as a way to trace their history.
• Deleting every unusual example automatically, which can remove valid rare cases or minority data.
• Trusting a checksum as proof that a model artifact was safe at its source.
• Changing production data or using private examples to reproduce a suspected backdoor.
• Treating the toy nearest-neighbor test as evidence about a neural network or commercial model.
• Reporting a successful trigger test without also reporting the control case and the scope of the experiment.

Recap and try it yourself

Data poisoning is deliberate influence over some of the examples or signals used to train or fine-tune a model. A backdoor is a hidden conditional behavior that appears when a trigger pattern is present. Retrieval contamination can affect model responses too, but it is a separate runtime path from changed training data. Data provenance, controlled writes, label review, trusted holdouts, version comparisons, artifact verification, and post-deployment monitoring make problems easier to prevent and investigate.

Try this in the local lab: run the clean and poisoned classifier tests, then read the training rows and identify which one carries the synthetic trigger and target label. Write down what the normal control tells you and what it cannot tell you. Sketch one approved data source your own project uses and list what provenance you would need to investigate a behavior change. Keep the exercise on synthetic or authorized data.