UAT, Go-Live, and Day 2 Operations

After this lesson, you will be able to plan user acceptance testing, make a release decision from agreed evidence, stage a go-live, and hand daily operation to named owners. NorthPort Freight is fictional; the workflow, people, and test results below are illustrative. This lesson reuses the Python acceptance runner from Evals as Acceptance Criteria and includes its complete code and tests so the example can be run on its own.

A successful demo answers whether something can work. User acceptance testing (UAT) asks whether the delivered system meets customer-approved needs in the environment and workflow where it will be used. Go-live is a controlled change that makes a service available to its intended users. Day 2 operations are the ongoing work after the initial release: monitoring, support, incident response, maintenance, access changes, and product or model updates. The FDE helps the customer move between these stages without treating a launch as the end of the work.

The customer owns acceptance and operational decisions for its service. The FDE can prepare evidence, identify risks, recommend a rollout, and help resolve defects. The FDE should not silently accept risk for the customer or declare a service operational when the responsible team has not agreed to own it.

Plan UAT around real tasks

Start with the acceptance criteria agreed during discovery and evaluation design. Translate each criterion into a task a representative user or operator can perform, the evidence they should observe, and the rule for passing. Include normal cases and important exceptions. A dispatcher might ask the assistant how to handle a familiar delivery exception, verify that the cited procedure is approved, and decide whether the response is useful in the middle of a shift. An operator might verify that the procedure index shows its last successful refresh and that an alert reaches the support queue.

UAT is not a substitute for engineering tests, security review, performance checks, or operational readiness. It provides evidence from users and customer owners about the behaviors they agreed to accept. Record which checks are automated, which need expert review, and which are outside the test. A passing sample cannot prove that every future answer is correct or that the service will never fail.

Agree the test scope with the customer before running it. Identify the release candidate, environment, dataset and criteria versions, user roles, test window, and who will observe and decide. Use approved test accounts and synthetic or authorized data. Confirm that UAT can run without affecting real dispatch decisions. Tell participants how to report a defect, what counts as a defect, and who can answer questions. Include users from relevant shifts and roles; a single manager's review may miss workflow details that appear only during real use.

Each test case should have a stable identifier, preconditions, task, expected observable result, evidence location, result, and reviewer. Keep failed and blocked cases in the record. “Blocked” means the team could not complete a check because a dependency, account, or environment was unavailable; it is not a pass. If a criterion is ambiguous, stop and resolve it with the approver before grading. Do not rewrite the expected behavior after seeing the candidate's result just to make the release pass.

For an AI feature, run the versioned acceptance dataset alongside human UAT. The automated runner can catch a missing citation identifier, empty output, missing case, or unapproved criteria version. A dispatcher or domain reviewer still needs to assess whether an answer is understandable, relevant, and supported by its cited procedure. Record disagreement and uncertainty instead of averaging them away. The same criteria and dataset should be used for the candidate comparison unless a reviewed change creates a new version.

Here is a small UAT record for the fictional NorthPort service:

The names and results are examples only. Before testing, the customer should decide how each outcome is judged and where evidence is stored. Avoid capturing more customer content than needed in screenshots or test notes.

Decide whether the release gate is open

Summarize UAT by criterion, not only by number of cases. Include the candidate and environment, criteria and dataset versions, passed, failed, blocked, or unknown checks, defects and severity, reviewer decisions, unresolved risks, and the customer approver. A result of “four of five passed” does not show whether the failed case was a required safety behavior or a minor formatting issue.

The acceptance runner marks required criteria as eligible only when their cases pass under the approved version. A failed required case or unknown evidence keeps the gate closed. An advisory criterion can fail without blocking the gate if that was agreed in advance. The runner does not replace customer sign-off. The customer owner records acceptance or a documented exception in the system the parties agreed to use. A typed approver name in a local script is only example metadata.

Make the release decision explicit:

• Go: required criteria passed, readiness checks have owners, and the customer approver authorizes the planned rollout.
• No-go: a required criterion failed, the evidence is incomplete, an operational dependency is unresolved, or the authorized owner has not approved.
• Go with an accepted exception: only if the customer process permits it, the authorized owner accepts the specific risk in writing, and mitigations and review date are recorded.

An exception does not turn a failed test into a pass. Preserve the test result and link the exception record. If customer policy forbids an exception for a particular condition, the launch stays blocked.

Release in stages and prepare rollback

A staged rollout exposes the new version to a limited audience or traffic slice before expanding access. The customer and platform owners choose the sequence, traffic limits, duration, health signals, and stop conditions. A rollout might begin with internal users, continue with a small dispatcher group, and expand to more shifts after the team reviews error rates and user feedback. These are possible patterns, not a prescribed percentage or schedule.

Before each stage, verify that the release artifact and configuration are the approved versions, access controls are correct, monitoring is visible to the operating team, and the support channel is staffed. Confirm that any required data migration or index refresh is complete and reversible or recoverable. Note the exact version being released and the version to restore if the rollout stops. If rollback has data consequences, document them before launch.

Define stop conditions in terms of observable signals. Examples could include a required acceptance check failing, a sustained increase in service errors, stale procedure data beyond the agreed limit, an access control issue, or a confirmed unsafe answer. The actual thresholds and response windows must come from the customer agreement and risk review. Avoid inventing a generic uptime threshold and presenting it as a standard.

Rollback means returning the service to a previously approved state or disabling the new capability. It may involve switching traffic, restoring a prior artifact, disabling a feature flag, or returning users to the existing manual workflow. Test the rollback path where practical. Name who is authorized to invoke it, what signal triggers action, where the procedure lives, and how users are informed. If a database or index migration cannot be reversed safely, say so and define a forward recovery plan with the owner.

Define Day 2 ownership before launch

The service needs a named operating model. A team or role should own each routine task and failure path. Agree who monitors alerts, handles first response, investigates model and retrieval behavior, updates procedures, approves access, deploys changes, and contacts vendors. Define support hours, severity categories, response expectations, escalation routes, and what the FDE team continues to own. These agreements belong in the customer’s operational records; this lesson does not create a service-level commitment.

Monitoring should answer questions the operating team can act on: Is the service available? Are requests failing or slowing down? Is retrieval returning approved sources? Is the index current? Are costs or usage outside the agreed range? Are there authorization denials or repeated tool errors? Link the runbook to the dashboards and alerts. Logs and traces may contain user text or sensitive data, so use the customer’s access and retention rules. The course lesson on Debugging and Observability covers customer-safe investigation details.

For a model-backed service, the model identifier is only one part of the version. Record the prompt or system instruction version, retrieval index and source set, application artifact, tool configuration, evaluation set, and relevant provider settings. A provider can change a model alias or service behavior; recheck current product terms and identifiers. Before changing a model or prompt in production, define a review and evaluation path, compare against the accepted baseline, and follow the customer’s change process. Do not update a production model simply because a new version is available.

An incident is an unplanned event that disrupts service or creates a credible risk to users, data, or operations. Follow the customer’s incident process. The first responder should know how to declare or route an incident, preserve relevant evidence, communicate status, and avoid unsafe experimentation in production. After recovery, record what happened, impact, timeline, contributing factors, resolution, and follow-up owners. A blameless review is intended to improve systems and practices, not to erase accountability for decisions.

Day 2 work includes updates and routine maintenance as well as incidents. Agree how the team reviews new procedures, refreshes or repairs the index, rotates credentials, assesses dependency changes, handles access requests, checks backups, and retires old versions. Set a regular service review cadence suitable for the system and customer. The point is to make maintenance owned and visible, not to add meetings for their own sake.

Use this support handover template

Complete this with the customer’s service owner and verify each link and contact before the FDE engagement ends.

1Service and production version: [name, version, environment]
2Go-live decision: [approver, date, approval record]
3Customer service owner: [team and contact path]
4FDE or vendor responsibilities after launch: [scope and end date]
5Support hours and severity route: [hours, queue, incident process]
6
7Operational owners
8- First response and triage: [team / coverage / procedure]
9- Application and deployment: [team / access request / runbook]
10- Model and provider settings: [team / change process]
11- Procedure source and index: [owner / refresh process]
12- Security and privacy review: [owner / escalation path]
13
14Readiness links
15- Current architecture and known limitations: [link]
16- Monitoring and alert dashboard: [link and access check]
17- Runbooks for deploy, rollback, refresh, and incidents: [links]
18- Current acceptance report and criteria version: [link]
19- Change and release record: [link]
20- Vendor support and status page: [approved links]
21
22Known issues and accepted exceptions
23- [issue, user impact, workaround, owner, review date]
24
25Upcoming maintenance and review
26- [change, owner, approval path, planned date]
27- Next service review: [date and participants]
28
29Handover rehearsal
30- Customer operator: [role or name]
31- Tasks rehearsed: [list]
32- Open questions and owners: [list]
33- Customer confirmation: [record in approved system]

Walk through an ordinary task and a failure scenario with the operator. Ask them to deploy or inspect a test version, locate a failed refresh alert, follow the runbook, and find the escalation route. Observe whether they can do it without relying on the FDE’s memory. Update the docs or access model when the rehearsal reveals a gap. Handover is a transfer of understanding and responsibility, not just a meeting invite and a folder of files.

The acceptance runner

This standard-library example is the same runner introduced in lesson 14. It evaluates a few synthetic fixtures; it does not call a model or judge semantic quality. It demonstrates a narrow gate: required criteria must pass, and missing approval or evidence remains unknown. Save the code as acceptance_runner.py.

Save this as acceptance_runner.py:

1from __future__ import annotations
2
3from dataclasses import dataclass
4from typing import Literal
5
6
7@dataclass(frozen=True)
8class Criterion:
9    criterion_id: str
10    description: str
11    required: bool
12    approved_by: str
13    criteria_version: str
14
15
16@dataclass(frozen=True)
17class Case:
18    case_id: str
19    criterion_id: str
20    expected_sources: frozenset[str]
21    response: str | None
22    cited_sources: frozenset[str]
23
24
25@dataclass(frozen=True)
26class CaseResult:
27    case_id: str
28    status: Literal["pass", "fail", "unknown"]
29    reason: str
30
31
32@dataclass(frozen=True)
33class CriterionResult:
34    criterion_id: str
35    status: Literal["pass", "fail", "unknown"]
36    passed: int
37    failed: int
38    unknown: int
39    cases: tuple[CaseResult, ...]
40
41
42@dataclass(frozen=True)
43class RunReport:
44    candidate_id: str
45    dataset_version: str
46    criteria_version: str
47    results: tuple[CriterionResult, ...]
48    release_allowed: bool
49
50
51def evaluate_case(case: Case) -> CaseResult:
52    """Check fixture completeness and whether an expected source is cited."""
53    if case.response is None:
54        return CaseResult(case.case_id, "unknown", "response was not evaluated")
55    if not case.response.strip():
56        return CaseResult(case.case_id, "fail", "response is empty")
57    if not case.expected_sources:
58        return CaseResult(case.case_id, "unknown", "no expected source is defined")
59    if not case.cited_sources.intersection(case.expected_sources):
60        return CaseResult(case.case_id, "fail", "no expected source was cited")
61    return CaseResult(case.case_id, "pass", "at least one expected source was cited")
62
63
64def run_acceptance(
65    candidate_id: str,
66    dataset_version: str,
67    criteria_version: str,
68    criteria: tuple[Criterion, ...],
69    cases: tuple[Case, ...],
70) -> RunReport:
71    """Score cases and open the gate only when every required criterion passes."""
72    if not candidate_id or not dataset_version or not criteria_version:
73        raise ValueError("candidate and version identifiers are required")
74    if not criteria:
75        raise ValueError("at least one approved criterion is required")
76
77    criterion_ids = {criterion.criterion_id for criterion in criteria}
78    if len(criterion_ids) != len(criteria):
79        raise ValueError("criterion identifiers must be unique")
80    if any(case.criterion_id not in criterion_ids for case in cases):
81        raise ValueError("every case must refer to a known criterion")
82
83    results: list[CriterionResult] = []
84    for criterion in criteria:
85        matching = tuple(case for case in cases if case.criterion_id == criterion.criterion_id)
86        if not criterion.approved_by.strip() or criterion.criteria_version != criteria_version:
87            case_results = tuple(
88                CaseResult(case.case_id, "unknown", "criterion approval is missing or version differs")
89                for case in matching
90            )
91            if not case_results:
92                case_results = (CaseResult("(criterion)", "unknown", "criterion approval is missing or version differs"),)
93        else:
94            case_results = tuple(evaluate_case(case) for case in matching)
95
96        passed = sum(result.status == "pass" for result in case_results)
97        failed = sum(result.status == "fail" for result in case_results)
98        unknown = sum(result.status == "unknown" for result in case_results)
99        if not matching and criterion.approved_by.strip() and criterion.criteria_version == criteria_version:
100            case_results = (CaseResult("(criterion)", "unknown", "criterion has no cases"),)
101            unknown = 1
102        status: Literal["pass", "fail", "unknown"]
103        if failed:
104            status = "fail"
105        elif unknown:
106            status = "unknown"
107        else:
108            status = "pass"
109        results.append(CriterionResult(
110            criterion.criterion_id, status, passed, failed, unknown, case_results
111        ))
112
113    by_id = {result.criterion_id: result for result in results}
114    release_allowed = all(
115        by_id[criterion.criterion_id].status == "pass"
116        for criterion in criteria
117        if criterion.required
118    )
119    return RunReport(
120        candidate_id, dataset_version, criteria_version, tuple(results), release_allowed
121    )
122
123
124def example() -> RunReport:
125    criteria = (
126        Criterion("C-01", "Cite an approved procedure", True, "Dispatch lead", "v2"),
127        Criterion("C-03", "Keep answers concise", False, "Product owner", "v2"),
128    )
129    cases = (
130        Case("case-001", "C-01", frozenset({"PROC-7"}), "Use the approved checklist. [PROC-7]", frozenset({"PROC-7"})),
131        Case("case-002", "C-01", frozenset({"PROC-9"}), "Check the dispatch notes. [PROC-2]", frozenset({"PROC-2"})),
132        Case("case-003", "C-03", frozenset({"PROC-7"}), "A concise answer. [PROC-7]", frozenset({"PROC-7"})),
133    )
134    return run_acceptance("candidate-0.8", "dispatch-cases-v3", "v2", criteria, cases)
135
136
137if __name__ == "__main__":
138    report = example()
139    for result in report.results:
140        print(f"{result.criterion_id}: {result.status} ({result.passed} pass, {result.failed} fail, {result.unknown} unknown)")
141    print(f"Release allowed: {report.release_allowed}")

Save the tests as test_acceptance_runner.py in the same directory:

1import unittest
2
3from acceptance_runner import Case, Criterion, evaluate_case, example, run_acceptance
4
5
6class AcceptanceRunnerTests(unittest.TestCase):
7    def setUp(self) -> None:
8        self.required = Criterion("C-01", "Cite approved source", True, "Dispatch lead", "v2")
9
10    def test_example_reports_failure_and_blocks_release(self) -> None:
11        report = example()
12        self.assertFalse(report.release_allowed)
13        self.assertEqual(report.results[0].status, "fail")
14        self.assertEqual(report.results[0].failed, 1)
15
16    def test_matching_source_passes(self) -> None:
17        case = Case("c1", "C-01", frozenset({"P-1"}), "Answer [P-1]", frozenset({"P-1"}))
18        self.assertEqual(evaluate_case(case).status, "pass")
19
20    def test_wrong_source_fails(self) -> None:
21        case = Case("c1", "C-01", frozenset({"P-1"}), "Answer [P-2]", frozenset({"P-2"}))
22        self.assertEqual(evaluate_case(case).status, "fail")
23
24    def test_blank_response_fails(self) -> None:
25        case = Case("c1", "C-01", frozenset({"P-1"}), "  ", frozenset({"P-1"}))
26        self.assertEqual(evaluate_case(case).status, "fail")
27
28    def test_missing_response_is_unknown(self) -> None:
29        case = Case("c1", "C-01", frozenset({"P-1"}), None, frozenset())
30        self.assertEqual(evaluate_case(case).status, "unknown")
31
32    def test_missing_expected_source_is_unknown(self) -> None:
33        case = Case("c1", "C-01", frozenset(), "Answer [P-1]", frozenset({"P-1"}))
34        self.assertEqual(evaluate_case(case).status, "unknown")
35
36    def test_required_unknown_blocks_release(self) -> None:
37        case = Case("c1", "C-01", frozenset({"P-1"}), None, frozenset())
38        report = run_acceptance("cand", "data-v1", "v2", (self.required,), (case,))
39        self.assertFalse(report.release_allowed)
40        self.assertEqual(report.results[0].status, "unknown")
41
42    def test_criteria_version_mismatch_is_unknown(self) -> None:
43        old = Criterion("C-01", "Cite source", True, "Dispatch lead", "v1")
44        case = Case("c1", "C-01", frozenset({"P-1"}), "Answer [P-1]", frozenset({"P-1"}))
45        report = run_acceptance("cand", "data-v1", "v2", (old,), (case,))
46        self.assertEqual(report.results[0].status, "unknown")
47        self.assertFalse(report.release_allowed)
48
49    def test_advisory_failure_does_not_block_gate(self) -> None:
50        advisory = Criterion("C-02", "Advisory", False, "Product owner", "v2")
51        cases = (
52            Case("required", "C-01", frozenset({"P-1"}), "Answer [P-1]", frozenset({"P-1"})),
53            Case("advisory", "C-02", frozenset({"P-1"}), "Answer [P-2]", frozenset({"P-2"})),
54        )
55        report = run_acceptance("cand", "data-v1", "v2", (self.required, advisory), cases)
56        self.assertTrue(report.release_allowed)
57        self.assertEqual(report.results[1].status, "fail")
58
59    def test_criterion_without_cases_is_unknown(self) -> None:
60        report = run_acceptance("cand", "data-v1", "v2", (self.required,), ())
61        self.assertEqual(report.results[0].status, "unknown")
62        self.assertFalse(report.release_allowed)
63
64    def test_duplicate_criterion_ids_are_rejected(self) -> None:
65        duplicate = Criterion("C-01", "Other", False, "Owner", "v2")
66        with self.assertRaises(ValueError):
67            run_acceptance("cand", "data-v1", "v2", (self.required, duplicate), ())
68
69    def test_unknown_criterion_reference_is_rejected(self) -> None:
70        case = Case("c1", "C-99", frozenset(), "Answer", frozenset())
71        with self.assertRaises(ValueError):
72            run_acceptance("cand", "data-v1", "v2", (self.required,), (case,))
73
74    def test_missing_run_identifier_is_rejected(self) -> None:
75        with self.assertRaises(ValueError):
76            run_acceptance("", "data-v1", "v2", (self.required,), ())
77
78
79if __name__ == "__main__":
80    unittest.main()

Run it with Python 3.10 or newer:

1python -m unittest -v
2python acceptance_runner.py

The example prints:

1C-01: fail (1 pass, 1 fail, 0 unknown)
2C-03: pass (1 pass, 0 fail, 0 unknown)
3Release allowed: False

The example intentionally blocks release because one required case cites the wrong procedure. The tests check pass, fail, unknown, missing approval version, missing cases, and invalid identifiers. Run them before using the example as a starting point, then ask what the code does not verify. It does not inspect the response text to prove that the cited source supports the recommendation, test a live model, exercise the production environment, or record formal customer acceptance. Those activities need appropriate human review, system tests, and customer processes. Lesson 14 explains how to turn an agreed behavior into an eval and why a bounded test result is not a guarantee about future use.

Mistakes beginners make

Treating UAT as a demo. A prepared walkthrough can show the happy path without testing the situations users face or the criteria the customer approved.

Fix: Give representative users realistic tasks, include important exception paths, and record evidence against the agreed criteria.

Counting blocked tests as passed. A missing account, unavailable environment, or absent approver does not provide evidence that the system works.

Fix: Keep blocked and unknown results visible. Resolve the dependency and rerun the check before a required gate opens.

Launching without an owner for rollback or support. An alert is not useful if nobody is responsible for responding, and a rollback plan is incomplete if no one can invoke it.

Fix: Name the authority, trigger, procedure, communication path, and follow-up record before rollout.

Calling handoff a folder transfer. A customer may receive runbooks but lack access, practice, or agreement on who handles model and data changes.

Fix: Rehearse ordinary and failure tasks with the operating team, verify access, and record the responsibilities they accept.

Recap and try it yourself

Use UAT to gather evidence from customer users and owners against criteria they approved. Keep failed, blocked, and unknown checks distinct. Make the go/no-go decision with the authorized customer owner, then release in stages with agreed health signals and a rehearsed rollback path. Before leaving, hand over named owners, support routes, dashboards, runbooks, version details, known issues, and a review process for model and data changes.

Try it yourself: use the fictional NorthPort cases above to draft a UAT plan with two dispatchers, a supervisor, the procedure owner, and a platform operator. Define what each must observe and what would block launch. Run the acceptance runner and explain why its failure closes the required gate. Then fill the support handover template with the operating owners and identify which entries still need customer confirmation.

Sources

Checked on 5 November 2026. GOV.UK's Service Standard on measuring success supports defining success measures and using data to improve services. Its API documentation guidance supports testing integration instructions and explaining sandbox limits. The Home Office's minimal documentation set supports handing over runbooks, observability, deployment details, incident processes, and named contacts. Google Cloud's operational readiness guidance supports roles, processes, monitoring, and planning for go-live and Day 2 operations; it is vendor guidance adapted here as a general planning aid. Python 3.12's unittest documentation supports the included standard-library suite. NorthPort and all cases are fictional and synthetic.