Compare Prompts with Graders

You have a task, success criteria, and a few saved test cases. Now you need a fair way to compare two prompt versions. Run both prompts against the same inputs, grade each answer using the same rules, and inspect where they fail. This lesson explains code, model, and human graders, and adds an offline comparison runner to the prompt lab.

The example outputs in this lesson come from small stub functions used to test the runner. They are not live model responses or evidence that one real prompt performs better.

Compare like with like

Suppose you have a baseline prompt and a revised prompt for summarizing delivery notes. If the baseline sees short, clear notes while the revision sees notes with missing times, the comparison tells you little. One prompt had an easier job.

Instead, run each version on every case in the same development set. Keep the graders identical. If the model can produce different outputs on repeated calls, run each prompt more than once on each case. A first experiment might use two prompt versions, ten cases, and three trials. That is 60 outputs to inspect or grade, so think about cost and time before making the set very large.

The number of model calls grows with the number of versions, cases, and trials. One prompt on 12 cases for three trials is 36 calls. Comparing two versions doubles that to 72. This repeated work helps reveal variability, but it does not guarantee a reliable statistical conclusion, especially with a tiny or unrepresentative case set.

This comparison follows Anthropic’s evaluation guidance: define what success means, run test cases, and use the results to refine the prompt. The steps apply across providers, though API details and available evaluation tools vary.

The AI Agents evaluation lesson uses the same basic ingredients: tasks, trials, graders, and reports. An agent evaluation may also need to check tool calls and the final state of an environment. Here we are comparing individual prompt outputs, so we can start with a smaller runner.

Choose the grader for the property

A grader is a check that assigns a result to an output. One answer can be checked by several graders, each for a different property. For a delivery summary, code might check that both times appear and that the reply is valid JSON. A person might judge whether its tone is respectful. A model with a rubric might help assess whether the summary is clear, though its judgment still needs review.

Code graders

Code graders are fast and repeatable when the property is precise. The prompt lab runner supports four simple checks:

These checks can also be too strict or too weak. Exact match rejects valid paraphrases. A contains check may pass an answer that says “not 8:32” because the characters are present. A regular expression may accept a date that does not exist. Valid JSON says nothing about whether the values are true. Code only checks the property you wrote down, so pair it with checks for meaning where needed.

Model graders

A model grader can apply a short rubric to a quality that is difficult to encode in ordinary code, such as whether a summary preserves the central point. Ask for a small set of possible verdicts: pass, fail, or unknown. The unknown option matters. If evidence is missing or the answer is ambiguous, the grader should be able to say so instead of guessing.

Keep the rubric narrow. “Is this response good?” combines too many judgments. “Does the response preserve both times and avoid claiming a cause that is not in the note?” is easier to inspect. Give the grader the original input, expected facts, answer, and rubric. Do not let it see which prompt version produced the answer if you are trying to reduce preference for one version.

Model graders can be inconsistent, miss subtle errors, or prefer longer and more confident answers. Check a sample of their verdicts yourself. If the grader calls an unsupported answer a pass, refine the rubric or use a source-based or code check for that fact. Do not treat a model score as ground truth merely because it is numeric.

Human review

People are useful for ambiguous cases, high-impact errors, and criteria that require real context. A reviewer can explain why an answer feels misleading in a way that a simple grader cannot. Human review is slower, so use it where the judgment matters. For a model grader, have a person review some pass, fail, and unknown examples to see whether the rubric is being applied sensibly.

You can combine graders. For example, require a code check for both times, then ask a person to judge tone. Keep the outcomes separate in the report so a failure is diagnosable. “Prompt B scored 7 out of 10” is less useful than “it preserved both times in 9 cases, but made an unsupported promise in 3.”

Run the prompt lab without a live model

The prompt lab’s comparison runner accepts prompt functions, test cases, and grader rules. A stub is a small substitute function that returns a saved response instead of calling a real model. Stubs make it possible to test whether the runner routes inputs and records grades correctly without API credentials, network calls, or unpredictable model output.

Save the runner below as prompt_lab_runner.py in your prompt-lab folder. It imports the loader from the previous lesson. It contains the code graders, the model-grader hook, the comparison loop and the report formatter:

prompt_lab_runner.py

1"""Run saved prompt variants against cases using code and optional model graders."""
2
3from collections import defaultdict
4from dataclasses import asdict, dataclass
5import json
6import re
7from typing import Any, Callable, Protocol
8
9from test_case_loader import TestCase
10
11
12class ModelJudge(Protocol):
13    def __call__(self, case: TestCase, output: str, rubric: str) -> str: ...
14
15
16@dataclass(frozen=True)
17class Grade:
18    verdict: str
19    reason: str
20
21
22def grade_code(output: str, check: dict[str, Any]) -> Grade:
23    """Run one deterministic check against an output string."""
24    kind = check.get("type")
25    value = check.get("value")
26    if kind == "exact":
27        passed = output.strip() == str(value)
28    elif kind == "contains":
29        passed = str(value) in output
30    elif kind == "regex":
31        passed = re.search(str(value), output) is not None
32    elif kind == "valid_json":
33        try:
34            json.loads(output)
35            passed = True
36        except (json.JSONDecodeError, TypeError):
37            passed = False
38    else:
39        raise ValueError(f"Unknown code grader: {kind!r}")
40    return Grade("pass" if passed else "fail", f"{kind} check")
41
42
43def grade_with_model(
44    case: TestCase, output: str, rubric: str, judge: ModelJudge
45) -> Grade:
46    """Use an injected judge and preserve its pass, fail, or unknown verdict."""
47    verdict = judge(case, output, rubric).strip().lower()
48    if verdict not in {"pass", "fail", "unknown"}:
49        return Grade("unknown", "model grader returned an invalid verdict")
50    return Grade(verdict, "model rubric")
51
52
53def run_comparison(
54    cases: list[TestCase],
55    prompts: dict[str, Callable[[str], str]],
56    code_checks: dict[str, list[dict[str, Any]]],
57    *,
58    model_rubrics: dict[str, str] | None = None,
59    model_judge: ModelJudge | None = None,
60    trials: int = 1,
61    split: str = "development",
62) -> dict[str, Any]:
63    """Run each prompt on each selected case and return records plus a report."""
64    if trials < 1:
65        raise ValueError("trials must be at least one")
66    selected = [case for case in cases if case.split == split]
67    if not selected:
68        raise ValueError(f"No cases in split {split!r}")
69    if not prompts:
70        raise ValueError("At least one prompt variant is required")
71    if any(not code_checks.get(case.id) and case.id not in (model_rubrics or {})
72           for case in selected):
73        raise ValueError("Every selected case needs at least one grader")
74    model_rubrics = model_rubrics or {}
75    if model_rubrics and model_judge is None:
76        raise ValueError("A model_judge is required when model rubrics are configured")
77
78    records = []
79    for name, prompt in prompts.items():
80        for case in selected:
81            for trial in range(1, trials + 1):
82                try:
83                    output = prompt(case.input)
84                    if not isinstance(output, str):
85                        raise TypeError("prompt variants must return text")
86                    grades = [grade_code(output, check)
87                              for check in code_checks.get(case.id, [])]
88                    if case.id in model_rubrics:
89                        assert model_judge is not None
90                        grades.append(grade_with_model(
91                            case, output, model_rubrics[case.id], model_judge
92                        ))
93                    verdicts = [grade.verdict for grade in grades]
94                    overall = (
95                        "fail" if "fail" in verdicts else
96                        "unknown" if "unknown" in verdicts else "pass"
97                    )
98                    records.append({
99                        "prompt": name, "case_id": case.id, "trial": trial,
100                        "output": output, "verdict": overall,
101                        "grades": [asdict(grade) for grade in grades],
102                    })
103                except Exception as error:
104                    records.append({
105                        "prompt": name, "case_id": case.id, "trial": trial,
106                        "output": None, "verdict": "fail", "grades": [],
107                        "error": f"{type(error).__name__}: {error}",
108                    })
109
110    grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
111    for record in records:
112        grouped[record["prompt"]].append(record)
113    summary = {}
114    for name, results in grouped.items():
115        total = len(results)
116        summary[name] = {
117            "pass_rate": sum(r["verdict"] == "pass" for r in results) / total,
118            "failures": sum(r["verdict"] == "fail" for r in results),
119            "unknown": sum(r["verdict"] == "unknown" for r in results),
120            "trials": total,
121        }
122    return {"split": split, "records": records, "summary": summary}
123
124
125def format_report(result: dict[str, Any]) -> str:
126    """Render a compact, stable text summary for terminal or saved reports."""
127    lines = [f"Prompt comparison ({result['split']} set)"]
128    for name, summary in result["summary"].items():
129        lines.append(
130            f"{name}: {summary['pass_rate']:.0%} pass, "
131            f"{summary['failures']} fail, {summary['unknown']} unknown "
132            f"across {summary['trials']} trials"
133        )
134    lines.append("Failures and unknowns:")
135    for record in result["records"]:
136        if record["verdict"] != "pass":
137            detail = record.get("error") or record["verdict"]
138            lines.append(
139                f"- {record['prompt']} / {record['case_id']} / "
140                f"trial {record['trial']}: {detail}"
141            )
142    return "\n".join(lines)

The runner receives each prompt as a function that takes the input text and returns an answer. The same function interface can wrap a real API call later, while tests can pass a stub. For the sample fixture, one code check requires that a response preserve 8:10 and another requires 8:32:

To try the runner yourself, save this as demo_compare.py in the prompt-lab folder. The two functions are stubs that return saved text, so no API key is needed:

1from prompt_lab_runner import format_report, run_comparison
2from test_case_loader import load_cases
3
4
5def baseline_prompt(text: str) -> str:
6    if "8:10" in text:
7        return "Predicted 8:10; arrived 8:32."
8    return "Predicted 9:00. Arrived 8:32."
9
10
11def revised_prompt(text: str) -> str:
12    if "8:10" in text:
13        return "Predicted 8:10; arrived 8:32."
14    return "Prediction unavailable; arrived 8:32."
15
16
17code_checks = {
18    "delivery-001": [
19        {"type": "contains", "value": "8:10"},
20        {"type": "contains", "value": "8:32"},
21    ],
22    "delivery-002": [
23        {"type": "contains", "value": "8:32"},
24        {"type": "contains", "value": "unavailable"},
25    ],
26}
27
28result = run_comparison(
29    cases=load_cases("cases.json"),
30    prompts={"baseline": baseline_prompt, "revised": revised_prompt},
31    code_checks=code_checks,
32    trials=3,
33    split="development",
34)
35print(format_report(result))

Run it with python3 demo_compare.py. The baseline stub guesses a prediction when the note shows none, and the second check for delivery-002 requires the word "unavailable", so the baseline fails there and the report lists the failing case. The revised stub passes.

For a live experiment, baseline_prompt and revised_prompt would call the model using your saved prompt text and the supplied input. The lesson’s tests instead use stubs. The runner records each answer and its grades, counts failures and unknowns, and creates a compact report. It does not decide whether your test cases are representative or whether your graders match the job. Those remain your responsibilities.

The model grader is passed as a function too. This keeps the runner independent of a provider. It accepts only pass, fail, or unknown; any other response becomes unknown rather than being silently treated as a pass. Connect a real model grader only after you have decided how to check its verdicts and handle invalid responses.

Read the results, then read the failures

Imagine a comparison returns this illustrative report:

1Prompt comparison (development set)
2baseline: 67% pass, 2 fail, 0 unknown across 6 trials
3revised: 83% pass, 1 fail, 0 unknown across 6 trials
4Failures and unknowns:
5- baseline / delivery-002 / trial 1: fail
6- baseline / delivery-002 / trial 2: fail
7- revised / delivery-001 / trial 2: fail

The revised prompt passed more graded outputs in this small run, but the report does not prove that it is better for every delivery note. Look at the failing answers. Did the revised prompt fix the missing-prediction case? Did it introduce a new problem in an ordinary case? Were the criteria applied correctly? A percentage tells you where to look; the examples help explain what happened.

If the two prompts tie, inspect the outputs and run more trials or cases only if the distinction matters. If a model grader returns unknown often, do not convert unknown to pass or silently drop it from the denominator. Find out whether the rubric, input, or answer lacks the evidence needed for a decision. For a safety-related criterion, an unknown can require human review.

Small samples are noisy. With six trials, one changed answer moves the rate by about 17 percentage points. Do not claim a durable improvement from a tiny difference. Add representative cases, repeat outputs if they vary, and gather more data for decisions with greater impact. A growing case set helps, but keep it balanced and preserve examples that were not used during prompt tuning.

Avoid tuning until the test set lies

When you read a failure, it is tempting to add a phrase that fixes that exact input. Sometimes that is the right change. But if you repeatedly tune against the same small set, the prompt can start matching those examples rather than solving the task broadly.

Keep a development set for diagnosing and improving the prompt. Use a separate holdout set for a later check, and do not look at it every time you edit. If you use a holdout failure to change the prompt, that example is now part of development. Save new unseen cases for the next holdout check. The lesson on success criteria and test cases explains how to build and maintain both groups.

Also resist the temptation to change the grader to make a favorite prompt pass. If a criterion is wrong or unclear, correct it deliberately and rerun every version under the same revised rule. Note the change. Otherwise your comparison mixes two changes: prompt behavior and the definition of success.

Tests for the runner

The prompt lab includes offline tests for the loader, runner, report, model-grader unknowns, and code checks. One test deliberately mutates a stub by removing a required arrival time. The contains grader must catch the missing fact. This is a small mutation check: it verifies that a test can detect at least one realistic regression rather than only passing a good answer.

Save the tests as prompt_lab_runner_test.py:

prompt_lab_runner_test.py

1"""Offline tests for prompt comparison, graders, unknowns, and mutations."""
2
3from pathlib import Path
4import unittest
5
6from prompt_lab_runner import (
7    format_report,
8    grade_code,
9    grade_with_model,
10    run_comparison,
11)
12from test_case_loader import load_cases
13
14
15ROOT = Path(__file__).parent
16CASES = load_cases(ROOT / "cases.json")
17CHECKS = {
18    "delivery-001": [
19        {"type": "contains", "value": "8:10"},
20        {"type": "contains", "value": "8:32"},
21    ],
22    "delivery-002": [{"type": "contains", "value": "8:32"}],
23}
24
25
26def reliable_stub(note):
27    if "did not show" in note:
28        return "The bus arrived at 8:32; no prediction was available."
29    return "The app predicted 8:10; the bus arrived at 8:32."
30
31
32def mutated_stub(note):
33    """Mutation: drop the actual time from the ordinary delivery response."""
34    if "did not show" in note:
35        return "The bus arrived at 8:32; no prediction was available."
36    return "The app predicted 8:10."
37
38
39class PromptLabRunnerTests(unittest.TestCase):
40    def test_compares_every_prompt_on_each_development_case_per_trial(self):
41        result = run_comparison(
42            CASES,
43            {"baseline": reliable_stub, "revised": reliable_stub},
44            CHECKS,
45            trials=3,
46        )
47        self.assertEqual(len(result["records"]), 12)
48        self.assertEqual(result["summary"]["baseline"]["pass_rate"], 1.0)
49
50    def test_holdout_is_only_run_when_requested(self):
51        result = run_comparison(CASES, {"baseline": reliable_stub}, {},
52                                model_rubrics={"delivery-003": "Do not guess."},
53                                model_judge=lambda *_: "unknown", split="holdout")
54        self.assertEqual([r["case_id"] for r in result["records"]], ["delivery-003"])
55        self.assertEqual(result["summary"]["baseline"]["unknown"], 1)
56
57    def test_rejects_missing_grader_instead_of_counting_vacuous_pass(self):
58        with self.assertRaisesRegex(ValueError, "at least one grader"):
59            run_comparison(CASES, {"baseline": reliable_stub}, {}, split="development")
60
61    def test_invalid_model_verdict_becomes_unknown(self):
62        grade = grade_with_model(CASES[0], "answer", "rubric", lambda *_: "maybe")
63        self.assertEqual(grade.verdict, "unknown")
64
65    def test_supported_code_graders(self):
66        checks = [
67            ("8:32", {"type": "exact", "value": "8:32"}, "pass"),
68            ("Bus: 8:32", {"type": "contains", "value": "8:32"}, "pass"),
69            ("Date: 2026-11-04", {"type": "regex", "value": r"\d{4}-\d{2}-\d{2}"}, "pass"),
70            ('{"time": "8:32"}', {"type": "valid_json"}, "pass"),
71        ]
72        for output, check, verdict in checks:
73            with self.subTest(kind=check["type"]):
74                self.assertEqual(grade_code(output, check).verdict, verdict)
75
76    def test_code_mutation_that_drops_a_required_fact_is_caught(self):
77        case = CASES[0]
78        mutated = mutated_stub(case.input)
79        grade = grade_code(mutated, {"type": "contains", "value": "8:32"})
80        self.assertEqual(grade.verdict, "fail")
81
82    def test_report_shows_failure_case_and_unknown_count(self):
83        result = run_comparison(
84            CASES, {"mutated": mutated_stub}, CHECKS, trials=1,
85        )
86        report = format_report(result)
87        self.assertIn("mutated / delivery-001", report)
88        self.assertIn("fail", report)
89
90
91if __name__ == "__main__":
92    unittest.main()

From inside the prompt-lab folder, run:

1python3 -m unittest discover -p '*_test.py'

The tests use no network and no model credentials. Passing them shows that this small runner handles its specified cases; it does not show that any real prompt performs well. For that you need real, reviewed examples and graders appropriate to the task.

Mistakes beginners make

1. Giving different prompts different inputs. Run every version on the same cases and use the same graders.
2. Using one exact string check for open-ended writing. Use exact checks for fixed values and a suitable rubric or human review for meaning and tone.
3. Treating valid JSON as a correct answer. Parse success only confirms syntax; validate the fields and facts too.
4. Forcing the model grader to choose. Keep an unknown verdict for unclear or unsupported judgments.
5. Looking only at the overall percentage. Read failures to see which behaviors changed and whether grading was fair.
6. Overinterpreting a small sample. One or two outputs can move a small pass rate substantially.
7. Tuning against holdout cases repeatedly. Once a holdout failure guides an edit, save a new unseen set.
8. Changing the rubric to favor a prompt. Correct criteria transparently and rerun all versions under the same rules.

Recap and try it yourself

Compare prompt versions using the same cases, graders, and number of trials. Use code for exact properties, model graders for narrow language judgments, and people for ambiguity or high-impact review. Keep unknown verdicts visible. Read individual failures before celebrating a percentage, and treat results from a small set as an early signal. Preserve holdout cases until you need a separate check.

Try this with the delivery-note fixture:

1. Write a baseline and a revised prompt for the same task.
2. Use at least one exact or contains check and one check that needs a person or model rubric.
3. Run both versions against every development case for three trials.
4. Read the failures and confirm that the grader is fair. Add one realistic case for a failure you did not cover.
5. Save the holdout case for after you finish tuning on the development set.

You have a repeatable comparison, not a verdict from one impressive output. The next lesson will show how to use the failure patterns to debug the prompt itself.