Evals as Acceptance Criteria

After this lesson, you will be able to turn a customer's expectations for an AI feature into measurable checks, run those checks against a fixed dataset, compare a new candidate with a baseline, and explain what the results do and do not prove. You will also build a small Python acceptance runner that blocks release when a required, customer-approved criterion fails. NorthPort Freight is fictional, and all records and outputs below are synthetic.

An AI demo can look convincing while leaving the hard questions unanswered. Did the system cite a current procedure? Did it admit when an answer was outside its approved knowledge? Will a prompt edit change behavior on cases that worked yesterday? A customer needs evidence tied to their task, not a confident impression from a handful of examples.

An evaluation, often shortened to eval, is a structured test of an AI system against a stated objective. It includes examples, a scoring method, and a rule for interpreting results. An acceptance criterion is a condition the customer agrees must hold before a delivery is accepted. The two overlap when an eval provides evidence for an acceptance criterion. Some requirements still need a person to judge them, and some can be checked with ordinary software tests. Avoid forcing every promise into a single score.

OpenAI's evaluation guidance describes a cycle of defining an objective, collecting a dataset, choosing metrics, running comparisons, and evaluating continuously. It also warns against generic measures and impression-based testing, and recommends calibrating automated scoring with human feedback. These are useful design principles across tools. The guide currently carries a deprecation notice for the legacy Evals platform, so check current platform status before choosing a vendor service. This lesson implements the underlying workflow locally and does not call a model or service.

Turn a requirement into a check

Start with the workflow and the person who owns it. NorthPort dispatchers ask what to do when a delivery exception occurs. The proposed assistant retrieves an approved dispatch procedure and drafts a response for a dispatcher to review. A statement such as "the assistant should be safe and accurate" is too broad to accept or reject. Ask what a dispatcher must be able to verify and what the system must do when it cannot.

Write a criterion so that two reviewers can tell what evidence would satisfy it. A useful record includes an identifier, a plain-language behavior, the examples that exercise it, the evaluator, the threshold, whether it blocks release, and the person who approved it. Define terms in the criterion itself. For example, "show the procedure ID in the answer" is more testable than "provide traceability." The customer should approve both the desired behavior and the meaning of a pass.

Here is an illustrative acceptance record for the fictional dispatch assistant:

These criteria are examples, not a real customer's signed approval. In practice, put the customer owner, approval date, version, and approval record beside the criteria. The sample runner represents approval as an explicit reviewer name and version. It does not create a digital or legal signature. Use the customer's approved record system when formal sign-off is required.

Before collecting examples, agree the boundary. Which procedure set is authoritative? What counts as an in-scope question? Is a refusal acceptable when retrieval is empty? Can an answer cite more than one procedure? Who can resolve a disagreement between a dispatcher and a product owner? These questions prevent a hidden assumption from becoming a test rule after the results arrive.

The dataset needs examples of ordinary use and plausible trouble. Include the common dispatch questions, ambiguous wording, missing source material, outdated procedures, conflicting instructions, and attempts to request an action outside the assistant's authority. Ask a domain expert to review the expected behavior. Synthetic cases are useful for development, but they cannot establish that the sample represents the customer's real traffic. Production examples need permission, minimization, access controls, and an agreed handling policy before use.

Keep the first dataset modest enough to inspect case by case. Each record should carry a stable case ID, the input, the expected behavior, and any approved source identifiers. Do not put sensitive customer information in a test fixture just because it makes the example feel realistic. Keep datasets separate from the evaluator code so reviewers can see what changed. Give the dataset its own version and owner.

Choose checks that match the promise

An exact check is useful when the expected behavior is crisp. The sample runner can verify that a cited source identifier belongs to the allowed set for that case, and that the response is not blank. This is deterministic: the same fixture produces the same result. It does not verify that the cited procedure actually supports every sentence. A real system needs additional review for that claim, such as expert annotation or a carefully validated evaluator.

A rubric is a written scoring guide that describes what acceptable, borderline, and failing responses look like. A human reviewer can apply one to questions such as whether a refusal is appropriate or an answer preserves an important qualification. Provide examples for each rating and train reviewers on the distinctions. If two reviewers often disagree, investigate the rubric or case before averaging their scores into a reassuring number.

An LLM-as-a-judge uses a model to score another model's output against instructions or a reference. It can help sort large batches, but it can also miss errors, favor a longer answer, or interpret a vague rubric inconsistently. Compare its scores with expert labels on representative examples before using it to make a release decision. Keep a human escalation path for safety-sensitive or disputed cases. Prompt Engineering's lessons on success criteria and test cases and comparing prompts with graders explore these design choices in more depth. The AI Agents lesson on evaluating AI agents covers checks for tool use and agent behavior.

Do not collapse every measure into one weighted score unless the customer understands and accepts the trade-off. A high average can hide a complete failure on a rare but consequential case. Keep hard release requirements separate from advisory measures. For example, a failed required citation criterion can block release even when an answer-length metric looks good. The gate should follow the signed requirements, not a convenient average.

Build a repeatable evaluation loop

An eval pipeline is the repeatable sequence that selects a dataset, runs a candidate, scores outputs, and stores a report. The useful output is not only a pass percentage. Keep the candidate identifier, prompt version, model identifier where relevant, dataset version, criteria version, evaluator version, run time, case-level outcomes, and known limitations. Those fields make the result interpretable later. Do not log secrets or full customer content without an approved reason and retention policy.

When a prompt changes, record the change as a new version and rerun the same dataset. Prompt versioning means assigning an identifiable revision to the instructions that control model behavior, along with enough context to reproduce what was evaluated. Keep the old version and report. Do not edit an old prompt in place and leave its previous result labelled as if it described the new behavior. The Prompt Engineering lesson on versioning and documenting prompts provides a broader prompt record template.

For a fair regression comparison, hold the dataset, criteria, evaluator, and relevant runtime settings constant while changing one candidate variable at a time. If the model or retrieval index changes too, record that as part of the candidate. Otherwise the comparison cannot tell which change caused a difference. When a criterion or dataset changes, make a new version and explain why. A stricter rubric can lower a score without any model behavior changing; that is a useful correction, but it is not evidence of a regression in the candidate.

The checks should run whenever a change could affect the behavior: prompt edits, model changes, retrieval updates, tool changes, or application code changes. A small local suite can run during development. A larger or more costly set can run before a release. New production incidents and customer feedback should create candidate cases, after review and any required redaction. The suite should grow around observed failures, not only the examples the team already knows how to pass.

Read results without overstating them

The sample acceptance runner reports pass and fail per criterion, with cases and reasons. A complete report should also distinguish a check that was not run from one that passed. "Unknown" is meaningful: a judge service may be unavailable, a human review may be pending, or a required data source may not have been approved. Treating unknown as pass silently waives the requirement. Treating it as fail may be safer for a hard release gate, but label the reason so the team can resolve it.

Read case-level failures before interpreting a rate. If one source identifier is missing, inspect whether retrieval failed, the expected answer is wrong, or the response omitted a valid citation. If a refusal case fails, inspect whether the case was truly outside scope and whether the rubric is clear. A number identifies where to look; it does not explain why the behavior occurred.

A test set is a sample of possible use, not a proof of all future behavior. It can miss rare phrasing, changing procedures, new workflows, distribution shifts, or interactions among tools. A passing result supports a bounded statement: this candidate passed these checks on this dataset under the recorded setup. It does not prove the system can never fail or act outside its intended behavior.

Report both the evidence and its boundary. A release note might say: "Candidate fde-assistant-0.8 passed the two required deterministic checks on dispatch-cases-v3 against criteria v2. Expert review of refusal quality is still pending, so that criterion remains unknown and the release gate is closed." This is more useful than "the eval score is 100 percent" because it identifies what ran and what remains unresolved.

The release gate should be designed with the customer before it is used. Decide which criteria block, who can accept a documented exception, what evidence is attached, and how a rollback decision is made. If the customer changes a requirement, revise and approve the criteria before grading a candidate against it. Preserve the previous report so both parties can see which agreement applied at that point in time.

The acceptance runner

The code below uses only Python's standard library. It scores a tiny fixed set of synthetic response fixtures; it does not generate responses, call a model, or verify semantic truth. Each criterion has a reviewer and version in its approval record. The runner refuses to issue a release decision if approval is missing, a required criterion fails, or a required criterion has no evaluated cases. Advisory failures appear in the report but do not block release. In a customer system, replace the in-file records with versioned, reviewed data and connect the approval metadata to the customer's sign-off process.

Save this as acceptance_runner.py:

1from __future__ import annotations
2
3from dataclasses import dataclass
4from typing import Literal
5
6
7@dataclass(frozen=True)
8class Criterion:
9    criterion_id: str
10    description: str
11    required: bool
12    approved_by: str
13    criteria_version: str
14
15
16@dataclass(frozen=True)
17class Case:
18    case_id: str
19    criterion_id: str
20    expected_sources: frozenset[str]
21    response: str | None
22    cited_sources: frozenset[str]
23
24
25@dataclass(frozen=True)
26class CaseResult:
27    case_id: str
28    status: Literal["pass", "fail", "unknown"]
29    reason: str
30
31
32@dataclass(frozen=True)
33class CriterionResult:
34    criterion_id: str
35    status: Literal["pass", "fail", "unknown"]
36    passed: int
37    failed: int
38    unknown: int
39    cases: tuple[CaseResult, ...]
40
41
42@dataclass(frozen=True)
43class RunReport:
44    candidate_id: str
45    dataset_version: str
46    criteria_version: str
47    results: tuple[CriterionResult, ...]
48    release_allowed: bool
49
50
51def evaluate_case(case: Case) -> CaseResult:
52    """Check fixture completeness and whether an expected source is cited."""
53    if case.response is None:
54        return CaseResult(case.case_id, "unknown", "response was not evaluated")
55    if not case.response.strip():
56        return CaseResult(case.case_id, "fail", "response is empty")
57    if not case.expected_sources:
58        return CaseResult(case.case_id, "unknown", "no expected source is defined")
59    if not case.cited_sources.intersection(case.expected_sources):
60        return CaseResult(case.case_id, "fail", "no expected source was cited")
61    return CaseResult(case.case_id, "pass", "at least one expected source was cited")
62
63
64def run_acceptance(
65    candidate_id: str,
66    dataset_version: str,
67    criteria_version: str,
68    criteria: tuple[Criterion, ...],
69    cases: tuple[Case, ...],
70) -> RunReport:
71    """Score cases and open the gate only when every required criterion passes."""
72    if not candidate_id or not dataset_version or not criteria_version:
73        raise ValueError("candidate and version identifiers are required")
74    if not criteria:
75        raise ValueError("at least one approved criterion is required")
76
77    criterion_ids = {criterion.criterion_id for criterion in criteria}
78    if len(criterion_ids) != len(criteria):
79        raise ValueError("criterion identifiers must be unique")
80    if any(case.criterion_id not in criterion_ids for case in cases):
81        raise ValueError("every case must refer to a known criterion")
82
83    results: list[CriterionResult] = []
84    for criterion in criteria:
85        matching = tuple(case for case in cases if case.criterion_id == criterion.criterion_id)
86        if not criterion.approved_by.strip() or criterion.criteria_version != criteria_version:
87            case_results = tuple(
88                CaseResult(case.case_id, "unknown", "criterion approval is missing or version differs")
89                for case in matching
90            )
91            if not case_results:
92                case_results = (CaseResult("(criterion)", "unknown", "criterion approval is missing or version differs"),)
93        else:
94            case_results = tuple(evaluate_case(case) for case in matching)
95
96        passed = sum(result.status == "pass" for result in case_results)
97        failed = sum(result.status == "fail" for result in case_results)
98        unknown = sum(result.status == "unknown" for result in case_results)
99        if not matching and criterion.approved_by.strip() and criterion.criteria_version == criteria_version:
100            case_results = (CaseResult("(criterion)", "unknown", "criterion has no cases"),)
101            unknown = 1
102        status: Literal["pass", "fail", "unknown"]
103        if failed:
104            status = "fail"
105        elif unknown:
106            status = "unknown"
107        else:
108            status = "pass"
109        results.append(CriterionResult(
110            criterion.criterion_id, status, passed, failed, unknown, case_results
111        ))
112
113    by_id = {result.criterion_id: result for result in results}
114    release_allowed = all(
115        by_id[criterion.criterion_id].status == "pass"
116        for criterion in criteria
117        if criterion.required
118    )
119    return RunReport(
120        candidate_id, dataset_version, criteria_version, tuple(results), release_allowed
121    )
122
123
124def example() -> RunReport:
125    criteria = (
126        Criterion("C-01", "Cite an approved procedure", True, "Dispatch lead", "v2"),
127        Criterion("C-03", "Keep answers concise", False, "Product owner", "v2"),
128    )
129    cases = (
130        Case("case-001", "C-01", frozenset({"PROC-7"}), "Use the approved checklist. [PROC-7]", frozenset({"PROC-7"})),
131        Case("case-002", "C-01", frozenset({"PROC-9"}), "Check the dispatch notes. [PROC-2]", frozenset({"PROC-2"})),
132        Case("case-003", "C-03", frozenset({"PROC-7"}), "A concise answer. [PROC-7]", frozenset({"PROC-7"})),
133    )
134    return run_acceptance("candidate-0.8", "dispatch-cases-v3", "v2", criteria, cases)
135
136
137if __name__ == "__main__":
138    report = example()
139    for result in report.results:
140        print(f"{result.criterion_id}: {result.status} ({result.passed} pass, {result.failed} fail, {result.unknown} unknown)")
141    print(f"Release allowed: {report.release_allowed}")

Save the tests as test_acceptance_runner.py in the same directory:

1import unittest
2
3from acceptance_runner import Case, Criterion, evaluate_case, example, run_acceptance
4
5
6class AcceptanceRunnerTests(unittest.TestCase):
7    def setUp(self) -> None:
8        self.required = Criterion("C-01", "Cite approved source", True, "Dispatch lead", "v2")
9
10    def test_example_reports_failure_and_blocks_release(self) -> None:
11        report = example()
12        self.assertFalse(report.release_allowed)
13        self.assertEqual(report.results[0].status, "fail")
14        self.assertEqual(report.results[0].failed, 1)
15
16    def test_matching_source_passes(self) -> None:
17        case = Case("c1", "C-01", frozenset({"P-1"}), "Answer [P-1]", frozenset({"P-1"}))
18        self.assertEqual(evaluate_case(case).status, "pass")
19
20    def test_wrong_source_fails(self) -> None:
21        case = Case("c1", "C-01", frozenset({"P-1"}), "Answer [P-2]", frozenset({"P-2"}))
22        self.assertEqual(evaluate_case(case).status, "fail")
23
24    def test_blank_response_fails(self) -> None:
25        case = Case("c1", "C-01", frozenset({"P-1"}), "  ", frozenset({"P-1"}))
26        self.assertEqual(evaluate_case(case).status, "fail")
27
28    def test_missing_response_is_unknown(self) -> None:
29        case = Case("c1", "C-01", frozenset({"P-1"}), None, frozenset())
30        self.assertEqual(evaluate_case(case).status, "unknown")
31
32    def test_missing_expected_source_is_unknown(self) -> None:
33        case = Case("c1", "C-01", frozenset(), "Answer [P-1]", frozenset({"P-1"}))
34        self.assertEqual(evaluate_case(case).status, "unknown")
35
36    def test_required_unknown_blocks_release(self) -> None:
37        case = Case("c1", "C-01", frozenset({"P-1"}), None, frozenset())
38        report = run_acceptance("cand", "data-v1", "v2", (self.required,), (case,))
39        self.assertFalse(report.release_allowed)
40        self.assertEqual(report.results[0].status, "unknown")
41
42    def test_criteria_version_mismatch_is_unknown(self) -> None:
43        old = Criterion("C-01", "Cite source", True, "Dispatch lead", "v1")
44        case = Case("c1", "C-01", frozenset({"P-1"}), "Answer [P-1]", frozenset({"P-1"}))
45        report = run_acceptance("cand", "data-v1", "v2", (old,), (case,))
46        self.assertEqual(report.results[0].status, "unknown")
47        self.assertFalse(report.release_allowed)
48
49    def test_advisory_failure_does_not_block_gate(self) -> None:
50        advisory = Criterion("C-02", "Advisory", False, "Product owner", "v2")
51        cases = (
52            Case("required", "C-01", frozenset({"P-1"}), "Answer [P-1]", frozenset({"P-1"})),
53            Case("advisory", "C-02", frozenset({"P-1"}), "Answer [P-2]", frozenset({"P-2"})),
54        )
55        report = run_acceptance("cand", "data-v1", "v2", (self.required, advisory), cases)
56        self.assertTrue(report.release_allowed)
57        self.assertEqual(report.results[1].status, "fail")
58
59    def test_criterion_without_cases_is_unknown(self) -> None:
60        report = run_acceptance("cand", "data-v1", "v2", (self.required,), ())
61        self.assertEqual(report.results[0].status, "unknown")
62        self.assertFalse(report.release_allowed)
63
64    def test_duplicate_criterion_ids_are_rejected(self) -> None:
65        duplicate = Criterion("C-01", "Other", False, "Owner", "v2")
66        with self.assertRaises(ValueError):
67            run_acceptance("cand", "data-v1", "v2", (self.required, duplicate), ())
68
69    def test_unknown_criterion_reference_is_rejected(self) -> None:
70        case = Case("c1", "C-99", frozenset(), "Answer", frozenset())
71        with self.assertRaises(ValueError):
72            run_acceptance("cand", "data-v1", "v2", (self.required,), (case,))
73
74    def test_missing_run_identifier_is_rejected(self) -> None:
75        with self.assertRaises(ValueError):
76            run_acceptance("", "data-v1", "v2", (self.required,), ())
77
78
79if __name__ == "__main__":
80    unittest.main()

Run it with Python 3.10 or newer:

1python -m unittest -v
2python acceptance_runner.py

The example prints:

1C-01: fail (1 pass, 1 fail, 0 unknown)
2C-03: pass (1 pass, 0 fail, 0 unknown)
3Release allowed: False

The test suite contains 13 tests. The first required criterion fails because the second answer cites a different procedure than the case expects. The advisory criterion passes, but it cannot compensate for that required failure. Notice that the evaluator only checks whether an expected identifier is present in the fixture's cited-source set. It does not parse the answer text or judge whether the procedure supports the recommendation. A production evaluator would need a controlled link between the output and its citations plus human review of semantic support.

Mistakes beginners make

Treating a score as proof. A metric summarizes the cases and rules supplied to it. It cannot reveal examples the dataset does not contain or assumptions the evaluator does not encode.

Fix: Report the exact candidate, dataset, criteria, and evaluator versions, then name what was not tested.

Letting a model grade an unclear rubric. An automated judge cannot resolve a disagreement that the customer has not defined. It may produce a precise-looking score while interpreting "good answer" differently from the dispatch team.

Fix: Ask the customer to approve observable behavior and example judgments. Compare automated scores with expert labels before using the judge for a gate.

Changing the test set to make a candidate pass. Removing a difficult case after a failure can erase useful regression evidence and hide the reason for the change.

Fix: Keep the prior dataset version. Add or revise cases through a documented review, then report the new version separately.

Treating unknown as success. Missing responses, unavailable graders, and absent reference labels do not satisfy a requirement.

Fix: Show unknown explicitly and decide before the run whether that state blocks the criterion. For required release criteria, keep the gate closed until evidence exists.

Calling a checklist a customer signature. A name typed into a local file is approval metadata, not proof of a legal signature or an approval in the customer's system.

Fix: Store the approval reference, approver, date, and criteria version in the system both parties agreed to use.

Recap and try it yourself

Acceptance criteria translate a customer's task into behaviors that can be reviewed and checked. Build a dataset around those behaviors, use evaluators that fit the claim, and keep required checks separate from advisory measures. Version the prompt, candidate, dataset, criteria, and evaluator so a later comparison has context. A release gate should open only when required criteria pass under the approved version; unknown evidence stays visible. The resulting report is evidence about a bounded test, not a guarantee about every future interaction.

Try it yourself: add a criterion for an out-of-scope question. Write three synthetic cases: one where the assistant should decline, one where it should answer from an approved procedure, and one where the reference label is missing. Decide with a fictional product owner which criterion blocks release and record the approval version. Extend the evaluator or test file to represent the refusal check, then run the suite. Before claiming it is ready for a customer, identify what the automated check cannot judge and what human review would still be needed.

Next, you will trace a bad agent run through logs and traces, and decide what evidence can be collected safely in a customer's environment.

Sources

Checked on 5 November 2026. OpenAI's evaluation best practices support task-specific objectives, datasets, metrics, repeated evaluation, and human calibration; its page currently notes deprecation of the legacy Evals platform. The OpenAI evals guide supports the workflow and evaluator distinctions. Python's unittest documentation for Python 3.12 supports the test suite. The internal lessons on criteria and test cases, graders, prompt versioning, and agent evaluation provide further study. NorthPort examples are fictional.