Adversarial Examples

After this lesson, you will be able to explain what an adversarial example is, distinguish an untargeted test from a targeted one, and measure a small input change against a toy classifier. You will also be able to plan a robustness test that states its assumptions and limits. The lab uses two fabricated numeric features and a fixed local model. It does not generate images, alter real data, or contact a deployed service.

A classifier turns an input into a category. A photo classifier might label an image as a bicycle or a bus; a text classifier might sort a message into “billing” or “returns.” The prediction depends on patterns the model learned. Sometimes a small change to an input can shift the model to a different category, even when the intended or human-understood example has not changed in a meaningful way. That changed input is called an adversarial example.

NIST's glossary entry defines an adversarial example as a modified test sample that causes a machine-learning model to misclassify or behave incorrectly at deployment time. Its Adversarial Machine Learning taxonomy places these examples in the study of evasion attacks, which happen when inputs are changed at inference time to alter a model's output. This is different from Lesson 11's data poisoning, where the training data or learning process is manipulated before or during model updates.

What makes a change adversarial?

The word “small” needs a definition. In an image, it might refer to a bounded change in pixel values. In another system, it might mean a limited change to a set of numeric features. A test may bound the largest per-feature change, the total change across features, or a domain-specific measure such as the number of modified tokens. That bound is part of the threat model, the assumptions about what an attacker can change and what outcome they want.

For image classification, research has shown cases where changes that are difficult for a person to notice can change a model's prediction. This does not mean every small pixel change fools every classifier, or that a model will see a transformed image the same way in every setting. Image scaling, compression, camera conditions, preprocessing, and model architecture all affect the result. A digital test should reflect the actual pipeline, including transformations applied before the model receives the image.

Text is different because words are discrete. You cannot usually nudge a word by 0.02 in the way a numeric pixel can be adjusted. A text robustness test may examine harmless typos, punctuation changes, formatting, or meaning-preserving paraphrases. But a replacement that changes a word can also change the meaning, and a model may be expected to react differently. So the test must define what should remain equivalent for the task. A changed classification by itself does not prove the new text is an adversarial example if the meaning or label legitimately changed.

The same caution applies to audio, video, structured records, and sensor inputs. A perturbation budget should match the representation and the real use case. If the system processes uploaded photos, a test on raw arrays may miss effects of resizing or compression. If the application classifies text after normalization, test the normalized end-to-end path. A useful result says which input representation and preprocessing steps it covered.

Untargeted and targeted tests

An untargeted evasion test asks whether a change can make the model return any incorrect class. If the expected result is “bus,” an untargeted failure might be any label other than “bus.” A targeted evasion test asks whether a change can make the model choose one specific incorrect class, such as “truck.” Targeted tests are narrower because they specify the desired wrong answer.

In a binary classifier, there is only one other class, so a successful untargeted change also happens to reach the targeted alternative. In a classifier with many classes, the difference matters. It is useful to report the goal clearly because “the model made a mistake” and “the model was made to choose this particular class” are different claims.

The test should also specify what the system owner expects. A model is not necessarily wrong just because its answer changed. If a person changes their mind, a document changes, or an image is cropped so the relevant object disappears, a different answer can be reasonable. For a robust test, the original and modified samples need an agreed expected label or task outcome. That reference may come from a trusted label, a domain reviewer, or a documented product rule.

Use a toy boundary to see the effect

The lab contains a two-feature linear classifier. Each input is a pair of numbers between zero and one. The model adds the two values and subtracts one. A score of zero or higher maps to class-a; a score below zero maps to class-b. This line is the model's decision boundary: points on one side get one category and points on the other side get the other category.

The clean synthetic sample is (0.54, 0.50). Its score is 0.04, so the model predicts class-a. For the exercise, that sample has a known expected class of class-a. The test then uses (0.49, 0.48), which changes the first feature by 0.05 and the second by 0.02. The largest per-feature change is 0.05, the exercise's budget. The new score is -0.03, so the prediction changes to class-b.

That largest per-feature distance is called the L-infinity distance, often written L∞. In plain language, it asks how much the most changed feature moved. The budget says no single feature may change by more than 0.05. This is one mathematical way to make “small” precise for this toy input. It is not a universal measure of smallness, and it does not say whether the changed features would be perceptually or semantically equivalent in a real task.

Save the lesson code

Save these 2 files in the lab folder you created in the safe lab lesson.

adversarial_examples_lab.py

1"""Two-feature linear classifier for a synthetic evasion example."""
2
3from __future__ import annotations
4
5from collections.abc import Sequence
6
7
8FEATURE_MIN = 0.0
9FEATURE_MAX = 1.0
10PERTURBATION_BUDGET = 0.05
11CLEAN_SAMPLE = (0.54, 0.50)
12NEARBY_NONFLIP_SAMPLE = (0.53, 0.50)
13PERTURBED_SAMPLE = (0.49, 0.48)
14EXPECTED_CLASS = "class-a"
15
16
17def score(features: Sequence[float]) -> float:
18    """Return a simple linear score for exactly two normalized features."""
19    if len(features) != 2:
20        raise ValueError("exactly two features are required")
21    if any(not FEATURE_MIN <= value <= FEATURE_MAX for value in features):
22        raise ValueError("features must be between zero and one")
23    return features[0] + features[1] - 1.0
24
25
26def predict(features: Sequence[float]) -> str:
27    """Assign a class based on which side of the toy boundary the input falls."""
28    return "class-a" if score(features) >= 0.0 else "class-b"
29
30
31def max_feature_change(original: Sequence[float], candidate: Sequence[float]) -> float:
32    """Measure the largest absolute change to any feature (L-infinity size)."""
33    if len(original) != len(candidate):
34        raise ValueError("original and candidate must have the same number of features")
35    if not original:
36        raise ValueError("at least one feature is required")
37    return max(abs(before - after) for before, after in zip(original, candidate))
38
39
40def is_within_budget(
41    original: Sequence[float],
42    candidate: Sequence[float],
43    budget: float = PERTURBATION_BUDGET,
44) -> bool:
45    """Check feature bounds and whether every coordinate stays in budget."""
46    if budget < 0:
47        raise ValueError("budget cannot be negative")
48    try:
49        score(original)
50        score(candidate)
51    except ValueError:
52        return False
53    # A tiny tolerance avoids rejecting a decimal budget because of binary
54    # floating-point representation (for example, 0.54 - 0.49).
55    return max_feature_change(original, candidate) <= budget + 1e-12

test_adversarial_examples.py

1"""Tests for a bounded, synthetic classifier evasion example."""
2
3import unittest
4
5from adversarial_examples_lab import (
6    CLEAN_SAMPLE,
7    EXPECTED_CLASS,
8    NEARBY_NONFLIP_SAMPLE,
9    PERTURBATION_BUDGET,
10    PERTURBED_SAMPLE,
11    is_within_budget,
12    max_feature_change,
13    predict,
14    score,
15)
16
17
18class AdversarialExampleTests(unittest.TestCase):
19    def test_clean_sample_matches_its_synthetic_expected_class(self) -> None:
20        self.assertEqual(predict(CLEAN_SAMPLE), EXPECTED_CLASS)
21
22    def test_small_bounded_change_flips_the_prediction(self) -> None:
23        self.assertEqual(predict(CLEAN_SAMPLE), "class-a")
24        self.assertEqual(predict(PERTURBED_SAMPLE), "class-b")
25        self.assertTrue(is_within_budget(CLEAN_SAMPLE, PERTURBED_SAMPLE))
26
27    def test_reported_maximum_change_matches_the_budget_example(self) -> None:
28        self.assertAlmostEqual(
29            max_feature_change(CLEAN_SAMPLE, PERTURBED_SAMPLE),
30            PERTURBATION_BUDGET,
31        )
32
33    def test_score_shows_samples_on_opposite_sides_of_boundary(self) -> None:
34        self.assertGreater(score(CLEAN_SAMPLE), 0.0)
35        self.assertLess(score(PERTURBED_SAMPLE), 0.0)
36
37    def test_change_larger_than_budget_is_not_accepted(self) -> None:
38        self.assertFalse(is_within_budget(CLEAN_SAMPLE, (0.48, 0.48)))
39
40    def test_bounded_change_that_stays_on_same_side_does_not_flip(self) -> None:
41        self.assertTrue(is_within_budget(CLEAN_SAMPLE, NEARBY_NONFLIP_SAMPLE))
42        self.assertEqual(predict(NEARBY_NONFLIP_SAMPLE), EXPECTED_CLASS)
43
44    def test_values_outside_feature_domain_are_rejected(self) -> None:
45        self.assertFalse(is_within_budget(CLEAN_SAMPLE, (1.01, 0.48)))
46
47    def test_classifier_rejects_wrong_shape_or_out_of_range_values(self) -> None:
48        with self.assertRaises(ValueError):
49            predict((0.5,))
50        with self.assertRaises(ValueError):
51            predict((0.5, 1.2))
52
53
54if __name__ == "__main__":
55    unittest.main()

Run the focused tests from the local lab directory:

1cd ai-red-teaming-lab
2python3 -m unittest -v test_adversarial_examples

The eight tests check that the clean point predicts its expected class, the bounded point crosses the decision boundary, a nearby bounded point remains on the same side, and a larger change or invalid feature value is not accepted as within the exercise budget. The budget comparison includes a small floating-point tolerance so decimal representation does not create a false failure. That implementation detail is ordinary numerical programming; it does not increase the model's robustness.

Then run all of the course's offline tests:

1python3 -m unittest -v

The before-and-after result illustrates a decision-boundary crossing. It is not proof that any particular classifier is vulnerable, and the input pair has no visual or real-world meaning. The expected class is part of the fabricated exercise. A real evaluation would need representative inputs, justified labels, a defined threat model, and the actual end-to-end system.

Design a robustness test

Before creating test inputs, write down the following:

1. System and model. Which classifier, model version, preprocessing steps, and application path are in scope?
2. Starting example. What input is authorized for testing, and what label or outcome should it receive?
3. Allowed changes. Which features may change, by how much, and what makes the changed example valid for the task?
4. Goal. Are you checking for any wrong class, a specific wrong class, or some other measurable behavior?
5. Success condition. What counts as a model failure, and how will you confirm that the input still represents the intended case?
6. Evidence. Which inputs, outputs, labels, model settings, and preprocessing details can be recorded safely and reproducibly?

This plan prevents a common measurement problem: changing an input so much that the correct answer is no longer known. In the toy example, the task label is set by the exercise. In a real image test, a reviewer may need to confirm that the object remains recognizable and that the label should stay the same. In a text test, a domain reviewer may need to decide whether a paraphrase still expresses the same request. Without that check, a changed label might be correct rather than a robustness failure.

Test ordinary examples as well as modified ones. A model that rejects every input would resist many evasion attempts, but it would not be useful. Record clean accuracy or another task-quality measure alongside the robustness result. Where inference is nondeterministic, repeat the same cases and report the run count and observed outcomes. Keep a baseline so that a defense can be checked for both robustness and unwanted changes to normal behavior.

If you test an image path, capture more than the model's final label when authorized: record the original sample identifier, transformation constraints, preprocessing, and expected label. If you test a text classifier, record why the changed text should preserve the original meaning. Keep examples synthetic or approved, and do not probe systems outside the written scope of your engagement.

When you summarize results, show the counts behind any percentage. For example, you might report how many valid modified examples were tested and how many still received the expected label. For a targeted test, report how many valid cases reached the specified target class. State what you excluded because its label or task meaning changed. These summaries describe only the chosen examples, model version, and threat model. A result from one small test set is not a universal robustness score.

Keep clean and modified outcomes separate. A model could classify the clean examples correctly but fail on permitted changes, or it could already classify the clean examples incorrectly. Those are different problems. Also report how often normal inputs were rejected or sent for review by a defense. A detector that catches modified examples but blocks many ordinary cases may have unacceptable product impact. Reviewers can then weigh the security result alongside accuracy, coverage, delay, and the cost of manual handling.

What the toy does not show

This lab uses a fixed linear decision rule. It does not learn from data, use a neural network, optimize inputs, process images, or probe a model through an API. A real adversarial-example assessment may involve a model owner who knows the architecture and parameters, an evaluator who can only submit queries, or a setting somewhere between those extremes. The amount of access changes the methods and the conclusions available to the tester.

The example also uses only two numeric features and two classes. Real data can have thousands or millions of features, complex preprocessing, multiple output categories, and task-specific constraints. An input change that is small under one mathematical distance may be obvious to a person or invalid for the task. Conversely, a seemingly tiny change may matter after preprocessing. Always connect the metric to the system and the user-facing meaning of its inputs.

Finally, one successful adversarial example says little about how common failures are. A robust evaluation needs a stated sample set, threat model, range of allowed modifications, success criteria, and enough runs to describe the observed behavior. Avoid claims such as “the model is robust” based on one test, or “the model is easy to fool” based on one cherry-picked example. Report the conditions under which the result occurred and the coverage you did not test.

Defenses and their limits

Adversarial training adds examples designed to challenge a model during training, with their correct labels, so the model learns to handle some modified inputs better. It can improve performance within the conditions represented by the training and evaluation process. It does not automatically cover new input types, stronger changes, or changes outside the modeled threat. If training and testing use nearly identical transformations, the result may overstate general robustness.

Other defenses may validate inputs, remove known noise, combine signals, detect unusual examples, or abstain when the model is uncertain. Abstention means the system declines to make a prediction and instead asks for review or uses a safer fallback. Each approach has tradeoffs. A detector can miss unfamiliar examples or flag normal ones; input cleanup can remove useful information; an ensemble can share weaknesses; and a fallback can add cost or delay.

Some methods aim to provide a certified robustness guarantee within a specific, formally defined perturbation bound. Such a guarantee can be valuable, but it applies only to the stated model, input representation, assumptions, and bound. It does not mean the system is safe against every attack or that the input itself is trustworthy. Read the guarantee's scope and test other risks separately.

NIST's taxonomy discusses adversarial training and other mitigation directions while emphasizing that defenses have limits and must be evaluated against meaningful threat models. A good test plan evaluates the defense under the same conditions an attacker is assumed to have, and includes stronger or adaptive tests when authorized. “Adaptive” here means the evaluation accounts for the defense rather than testing only a fixed, known example that the defense may have memorized.

Mistakes beginners make

• Treating every classification change as an adversarial failure, even when the input's meaning or correct label changed.
• Saying an input is “small” without naming the distance measure, budget, or domain constraint.
• Reporting a targeted result when the test only showed an arbitrary wrong class, or vice versa.
• Testing a model in isolation when the deployed application resizes, normalizes, filters, or otherwise transforms inputs first.
• Assuming image perturbations and text changes can be measured or interpreted the same way.
• Focusing only on robustness and ignoring whether the defense blocks ordinary useful inputs.
• Calling a defense strong because it defeats a few fixed cases chosen before the defense was built.
• Treating the two-feature toy model as evidence about real image, text, or hosted classifiers.
• Publishing robustness claims without the model version, test conditions, labels, and coverage limits.

Recap and try it yourself

An adversarial example is a modified test input that makes a model misclassify or behave incorrectly at deployment time. Evasion changes happen at inference, unlike poisoning that changes training inputs. An untargeted test seeks any wrong class; a targeted test seeks a specified wrong class. A useful robustness test defines the system, expected label, permitted changes, budget, goal, and evidence, then checks normal task quality as well as failures. Defenses such as adversarial training, detection, and certified methods can help within stated limits; none makes every system safe by itself.

Try this in the lab: run the clean and perturbed feature pairs, calculate the largest feature change, and record the scores on both sides of the boundary. Then change one feature by an amount that stays within the same budget but does not cross the boundary. Explain why that case is a useful negative control. Keep the work local and synthetic, and be clear that the model's artificial features do not represent a real image or text task.