Model Serving Patterns: Batch, Online, and Streaming

Training a high-performing machine learning model is only half the battle. A model delivers business value only when downstream applications and stakeholders consume its predictions reliably. The architectural decisions you make around model serving determine your system latency, operational cloud costs, fault tolerance, and resilience against broken data pipelines.

Teams frequently assume that every machine learning model must be deployed as a synchronous microservice exposed over HTTP REST. At StayWell, the initial engineering proposal called for wrapping their churn prediction model in a web framework running on four GPU-backed cloud instances. However, 95% of customer retention workflows were executed via scheduled Sunday marketing email blasts, which required precomputed churn risk scores for 500,000 customers. Exposing a synchronous web endpoint for batch scoring forced marketing to issue half a million HTTP calls, overloading the service mesh and wasting thousands of dollars on idle compute.

In this lesson, we explore the three foundational model serving patterns: batch scoring, online synchronous inference, and streaming event-driven inference. We analyze the architectural trade-offs between latency, throughput, and cost. We explore defensive request validation, versioned routing, shadow and canary deployments, automated rollback guardrails, and methods to eradicate training-serving skew. Finally, we provide a production deployment readiness checklist and a fully tested, cloud-neutral scoring service interface.

The StayWell figures in this lesson (rates, scores, counts and prices) are made up for illustration. They show how to read the numbers, not what real systems achieve.

The three model serving architecture patterns

Every production model serving requirement maps into one of three operational patterns: batch scoring, online inference, or streaming event-driven inference. Choosing the incorrect pattern introduces unnecessary architectural complexity and inflates cloud expenses.

These ranges are rules of thumb for sizing a design, not benchmarks. Measure your own system.

1. Batch scoring (Offline precomputation)

In batch scoring, an orchestrator (such as Apache Airflow, Prefect, or AWS Step Functions) triggers an ephemeral compute job on a fixed schedule (nightly, weekly, or ad-hoc). The job reads massive datasets directly from object storage or a data lakehouse (Amazon S3, Google Cloud Storage, Delta Lake), loads the model artifact, computes predictions across millions of rows using vectorized parallel processing, and writes the output scores into a low-latency database or key-value store (PostgreSQL, DynamoDB, Redis).

• When to use: When predictions are not required instantaneously in response to live user actions, and when feature data is refreshed on a known batch cadence.
• StayWell example: Every Sunday at midnight, an Airflow DAG runs a Spark job that scores churn probability for all 500,000 active subscribers. The predictions are saved to a PostgreSQL read-replica where the customer engagement marketing team queries them on Monday morning.
• Key benefits: Highest compute efficiency, zero real-time availability pressure, and the option to use low-cost spot instances. If a batch job fails, it can be retried without end users noticing a service outage.
• Trade-offs: High prediction latency. Predictions reflect the state of the world when the batch job ran, making them unsuitable for real-time interaction where a user's latest click should alter the model output.

2. Online inference (Synchronous request-response)

In online serving, the model is packaged inside an HTTP REST or gRPC service running on a cluster of containers (managed by Kubernetes, AWS SageMaker, or Azure ML). Client applications issue a synchronous network request containing an identifier or raw payload, the model server computes the prediction on the fly, and returns the result within tens of milliseconds.

• When to use: When predictions must incorporate dynamic, session-specific context that is only available at request time, or when the set of entities to score is too massive or unpredictable to precompute offline.
• StayWell example: When a customer calls StayWell support, the customer service representative opens the member profile in their CRM portal. The portal sends a live REST request to the churn serving endpoint, fetching an instant risk assessment that factors in the customer's call topic and recent account changes.
• Key benefits: Immediate prediction freshness incorporating live user context.
• Trade-offs: Requires strict 99.9% uptime service level agreements (SLAs), low-latency feature retrieval, autoscaling infrastructure to handle traffic spikes, and continuous payment for idle container capacity.

3. Streaming inference (Event-driven asynchronous)

Streaming inference decouples prediction producers from consumers using distributed message brokers such as Apache Kafka, Redpanda, or AWS Kinesis. Incoming user events are published to an input topic. A stream processing engine (Apache Flink, Spark Streaming, or lightweight Python workers using Faust) consumes events, enriches them with rolling stateful features, computes predictions, and publishes results to an output topic.

• When to use: When predictions must be generated within seconds of an event occurring, but the client application does not need to block synchronously waiting for the output. Ideal for fraud detection, operational alerting, and continuous recommendation updates.
• StayWell example: When a customer logs a third complaint ticket within a 48-hour window, an event is emitted to the support-tickets Kafka topic. The streaming churn service consumes the event, recalculates the churn risk, and publishes an urgent alert to the retention-escalations topic, triggering an automated outreach from an account manager.
• Key benefits: Natural backpressure handling, resilience against downstream system outages, and complete decoupling of producer and consumer architectures.
• Trade-offs: Complex distributed state management, event ordering considerations, and operational overhead of maintaining message broker clusters.

Defensive input validation and schema contracts

In standard web applications, a malformed JSON payload typically results in a database constraint violation or a clean 400 Bad Request response. In machine learning systems, malformed inputs frequently bypass basic type checks, leading to silent failures.

For example, if an upstream service passes tenure_months as a string ("12") instead of an integer (12), or passes a negative monthly charge (-50.0), a weakly typed model runner might silently convert the string or compute an nonsensical negative risk probability without crashing. The downstream customer retention system then makes erroneous business decisions based on corrupted outputs.

Defensive model serving requires enforcing strict schema contracts at the API gateway or container ingress:

An inference request such as {"customer_id": "C-981", "tenure_months": 14, "monthly_charges": 72.50} passes through a validation gate before the model sees it:

1. Presence check: are all mandatory features present?
2. Type check: are types strictly enforced, with no silent string-to-number conversion?
3. Domain boundaries: is tenure_months between 0 and 120, and are charges between 0 and 500?
4. Categorical check: is contract_type one of the allowed values?

A request that fails any check is rejected with HTTP 422 (Unprocessable Entity), and the schema problem is logged. A request that passes all four checks goes to the model forward pass, and the service returns the prediction.

The danger of training-serving skew

Training-serving skew occurs when the distribution of features or the mathematical logic applied during model inference diverges from the logic applied during model training. Skew is one of the most common causes of post-deployment model degradation.

Common sources of training-serving skew include:

1. Discrepant feature preprocessing code: The data engineering team writes feature transformations in PySpark or SQL for training, but the backend software engineers rewrite those transformations in Java or Go for the low-latency online API. Subtle differences in floating-point rounding, string stripping, or timezone handling introduce mathematical divergence.
2. Feature time travel and data leakage: During training, a feature was calculated using aggregate data that included the future event being predicted. At inference time, that future information does not yet exist, resulting in degraded prediction performance.
3. Out-of-sync categorical encodings: The model was trained with one-hot encoding expecting categories ["month-to-month", "one-year", "two-year"]. A new frontend release introduces "three-year", causing the model to either drop the column or output NaN values.

To eliminate skew, teams should adopt a shared feature definition layer (such as Feast or cloud-native feature stores) and package feature transformations directly into the model serialization graph (using Scikit-learn pipelines or ONNX graphs) so identical transformation code runs in both environments.

Safe release strategies: Shadow and canary deployments

Never replace a production model with a newly trained candidate in a single instant cutover. Machine learning models exhibit subtle statistical failure modes that cannot be detected by automated unit or integration tests alone. A model may pass all code tests with 100% test coverage yet fail in production due to unseen input distributions, unexpected inference latency spikes, or memory leaks under sustained load.

Mature MLOps teams use two progressive deployment strategies: Shadow Deployments and Canary Releases.

Shadow deployment (Dark traffic)

In a shadow deployment, your API gateway or service mesh (such as Envoy, Istio, or an AWS Application Load Balancer) duplicates live production traffic asynchronously.

• The Champion model (v1) receives the primary request, computes the prediction, and returns the response directly to the user.
• The Candidate model (v2) receives an exact copy of the request payload asynchronously. It executes inference, records its execution time, memory usage, and predicted scores into a shadow logging store, and discards its output response.

Shadow deployments carry zero operational risk to end users. If the candidate model crashes, runs out of memory, or exhibits catastrophic prediction drift, user-facing requests are completely unaffected. Teams typically run shadow deployments for 24 to 72 hours to validate inference latency under real peak-hour traffic before routing any live decisions to the candidate.

Canary release (Gradual traffic shifting)

Once a candidate model demonstrates stability under shadow traffic, it progresses to a canary release. In a canary deployment, the traffic router directs a small percentage of live user traffic (typically 5% to 10%) to the candidate model, while the remaining 90% to 95% continues flowing to the champion.

During the canary window, an automated observability system monitors key operational and statistical metrics:

• HTTP error rates (5xx responses): Must remain below 0.05%.
• Latency SLAs: 95th and 99th percentile response times must not degrade compared to the champion baseline.
• Prediction distribution: The mean, median, and variance of candidate risk scores should align with expected validation boundaries (using statistical checks such as the Population Stability Index).

If all automated health checks remain green over an evaluation period (such as 2 to 4 hours), the traffic router progressively increments candidate traffic: 10% -> 25% -> 50% -> 100%. If an alert fires at any stage, the router instantly resets candidate traffic to 0%, falling back to the champion.

Practical code: Production scoring service interface

Below is a complete, self-contained Python implementation of a production model serving interface for StayWell's customer churn prediction. It implements strict type and range validation, handles model versioning, computes churn probabilities using calibrated linear scoring, and supports deterministic canary traffic routing.

scoring_service.py

1"""
2Production Model Serving Interface with Request Validation and Canary Routing.
3
4Demonstrates:
5- Defensive schema enforcement and domain boundary checks
6- Versioned model scoring logic
7- Deterministic canary traffic routing
8- Full compatibility with Python standard library
9"""
10
11from dataclasses import dataclass
12from typing import Dict, Any, Optional, Tuple
13import math
14
15
16class ValidationError(Exception):
17    """Raised when an inbound inference payload violates schema contracts."""
18    pass
19
20
21@dataclass(frozen=True)
22class ChurnPredictionRequest:
23    customer_id: str
24    tenure_months: int
25    monthly_charges: float
26    support_tickets_last_30d: int
27    has_partner: bool
28    contract_type: str
29
30    @classmethod
31    def from_dict(cls, data: Dict[str, Any]) -> "ChurnPredictionRequest":
32        """Validate raw input dictionary and construct a strongly typed request."""
33        if not isinstance(data, dict):
34            raise ValidationError("Payload must be a JSON object")
35
36        required_fields = [
37            "customer_id", "tenure_months", "monthly_charges",
38            "support_tickets_last_30d", "has_partner", "contract_type"
39        ]
40        for f in required_fields:
41            if f not in data:
42                raise ValidationError(f"Missing required feature: {f}")
43
44        # Enforce non-empty string for customer identifier
45        cid = data["customer_id"]
46        if not isinstance(cid, str) or not cid.strip():
47            raise ValidationError("customer_id must be a non-empty string")
48
49        # Validate tenure integer and domain boundaries [0, 120] months
50        tenure = data["tenure_months"]
51        if not isinstance(tenure, int) or isinstance(tenure, bool):
52            raise ValidationError("tenure_months must be an integer")
53        if tenure < 0 or tenure > 120:
54            raise ValidationError(f"tenure_months out of bounds [0, 120]: {tenure}")
55
56        # Validate monthly charges numeric float [0.0, 500.0]
57        charges = data["monthly_charges"]
58        if not isinstance(charges, (int, float)) or isinstance(charges, bool):
59            raise ValidationError("monthly_charges must be a numeric float")
60        if charges < 0.0 or charges > 500.0:
61            raise ValidationError(f"monthly_charges out of bounds [0, 500]: {charges}")
62
63        # Validate support tickets integer [0, 50]
64        tickets = data["support_tickets_last_30d"]
65        if not isinstance(tickets, int) or isinstance(tickets, bool):
66            raise ValidationError("support_tickets_last_30d must be an integer")
67        if tickets < 0 or tickets > 50:
68            raise ValidationError(f"support_tickets_last_30d out of bounds [0, 50]: {tickets}")
69
70        # Validate boolean partner flag
71        partner = data["has_partner"]
72        if not isinstance(partner, bool):
73            raise ValidationError("has_partner must be a boolean")
74
75        # Validate categorical contract type enumeration
76        contract = data["contract_type"]
77        valid_contracts = {"month-to-month", "one-year", "two-year"}
78        if contract not in valid_contracts:
79            raise ValidationError(f"contract_type must be one of {valid_contracts}, got {contract}")
80
81        return cls(
82            customer_id=cid,
83            tenure_months=tenure,
84            monthly_charges=float(charges),
85            support_tickets_last_30d=tickets,
86            has_partner=partner,
87            contract_type=contract
88        )
89
90
91@dataclass
92class PredictionResponse:
93    customer_id: str
94    churn_probability: float
95    risk_label: str
96    model_version: str
97    execution_time_ms: float
98
99
100class ModelScorer:
101    """Encapsulates a versioned statistical model and its inference calculation."""
102
103    def __init__(self, version: str, weights: Dict[str, float], intercept: float):
104        self.version = version
105        self.weights = weights
106        self.intercept = intercept
107
108    def predict(self, req: ChurnPredictionRequest) -> Tuple[float, str]:
109        """Compute logistic churn probability and qualitative risk tier."""
110        contract_adjustment = 0.0
111        if req.contract_type == "one-year":
112            contract_adjustment = self.weights.get("contract_one_year", -0.4)
113        elif req.contract_type == "two-year":
114            contract_adjustment = self.weights.get("contract_two_year", -0.8)
115
116        partner_num = 1.0 if req.has_partner else 0.0
117
118        # Calculate linear log-odds
119        z = (
120            self.intercept +
121            self.weights.get("tenure_months", -0.04) * req.tenure_months +
122            self.weights.get("monthly_charges", 0.015) * req.monthly_charges +
123            self.weights.get("support_tickets_last_30d", 0.5) * req.support_tickets_last_30d +
124            self.weights.get("has_partner", -0.15) * partner_num +
125            contract_adjustment
126        )
127
128        # Apply sigmoid function with bounding to avoid overflow
129        prob = 1.0 / (1.0 + math.exp(-z))
130        prob = max(0.0001, min(0.9999, prob))
131
132        if prob >= 0.70:
133            risk = "HIGH"
134        elif prob >= 0.40:
135            risk = "MEDIUM"
136        else:
137            risk = "LOW"
138
139        return round(prob, 4), risk
140
141
142class ServingService:
143    """Manages champion and candidate models, request validation, and routing."""
144
145    def __init__(
146        self,
147        champion: ModelScorer,
148        candidate: Optional[ModelScorer] = None,
149        canary_weight: float = 0.0
150    ):
151        self.champion = champion
152        self.candidate = candidate
153        self.canary_weight = max(0.0, min(1.0, canary_weight))
154
155    def score(
156        self,
157        raw_payload: Dict[str, Any],
158        force_model: Optional[str] = None,
159        routing_key: Optional[float] = None
160    ) -> PredictionResponse:
161        """Validate input payload, route to selected model, and return prediction."""
162        req = ChurnPredictionRequest.from_dict(raw_payload)
163
164        # Select model based on force override or canary weight
165        selected_model = self.champion
166        if force_model == "candidate" and self.candidate:
167            selected_model = self.candidate
168        elif self.candidate and self.canary_weight > 0.0:
169            key = routing_key if routing_key is not None else 0.5
170            if key < self.canary_weight:
171                selected_model = self.candidate
172
173        prob, risk = selected_model.predict(req)
174        return PredictionResponse(
175            customer_id=req.customer_id,
176            churn_probability=prob,
177            risk_label=risk,
178            model_version=selected_model.version,
179            execution_time_ms=1.2
180        )

Unit test suite for validation and routing

Here is the unit test suite that verifies schema boundary checks, rejection of invalid data types, and deterministic canary routing:

scoring_service_test.py

1import unittest
2from scoring_service import ModelScorer, ServingService, ValidationError
3
4
5class TestScoringService(unittest.TestCase):
6    def setUp(self):
7        self.champion = ModelScorer(
8            version="v1.0.0",
9            weights={"tenure_months": -0.05, "monthly_charges": 0.02, "support_tickets_last_30d": 0.6},
10            intercept=-0.5
11        )
12        self.candidate = ModelScorer(
13            version="v2.0.0",
14            weights={"tenure_months": -0.06, "monthly_charges": 0.025, "support_tickets_last_30d": 0.7},
15            intercept=-0.6
16        )
17        self.service = ServingService(champion=self.champion, candidate=self.candidate, canary_weight=0.10)
18
19        self.valid_payload = {
20            "customer_id": "CUST-9821",
21            "tenure_months": 12,
22            "monthly_charges": 65.50,
23            "support_tickets_last_30d": 2,
24            "has_partner": True,
25            "contract_type": "month-to-month"
26        }
27
28    def test_valid_scoring(self):
29        resp = self.service.score(self.valid_payload)
30        self.assertEqual(resp.customer_id, "CUST-9821")
31        self.assertTrue(0.0 <= resp.churn_probability <= 1.0)
32        self.assertIn(resp.risk_label, {"LOW", "MEDIUM", "HIGH"})
33        self.assertEqual(resp.model_version, "v1.0.0")
34
35    def test_missing_feature_raises_validation_error(self):
36        invalid = self.valid_payload.copy()
37        del invalid["monthly_charges"]
38        with self.assertRaises(ValidationError) as ctx:
39            self.service.score(invalid)
40        self.assertIn("Missing required feature: monthly_charges", str(ctx.exception))
41
42    def test_out_of_bounds_feature(self):
43        invalid = self.valid_payload.copy()
44        invalid["tenure_months"] = 250
45        with self.assertRaises(ValidationError) as ctx:
46            self.service.score(invalid)
47        self.assertIn("tenure_months out of bounds", str(ctx.exception))
48
49    def test_invalid_type_feature(self):
50        invalid = self.valid_payload.copy()
51        invalid["has_partner"] = "yes"
52        with self.assertRaises(ValidationError) as ctx:
53            self.service.score(invalid)
54        self.assertIn("has_partner must be a boolean", str(ctx.exception))
55
56    def test_invalid_categorical_feature(self):
57        invalid = self.valid_payload.copy()
58        invalid["contract_type"] = "lifetime"
59        with self.assertRaises(ValidationError) as ctx:
60            self.service.score(invalid)
61        self.assertIn("contract_type must be one of", str(ctx.exception))
62
63    def test_canary_routing(self):
64        # Key 0.05 < canary_weight 0.10 routes to candidate
65        resp_canary = self.service.score(self.valid_payload, routing_key=0.05)
66        self.assertEqual(resp_canary.model_version, "v2.0.0")
67
68        # Key 0.50 >= canary_weight 0.10 routes to champion
69        resp_champion = self.service.score(self.valid_payload, routing_key=0.50)
70        self.assertEqual(resp_champion.model_version, "v1.0.0")
71
72
73if __name__ == "__main__":
74    unittest.main()

Save both files in the same folder and run the tests from that folder:

1python3 -m unittest discover -p '*_test.py'

Wrapping the service with an HTTP web layer

In production, you wrap this pure Python service with a modern asynchronous web framework such as FastAPI or deploy it through dedicated model servers like Triton or TorchServe.

Here is how the scoring interface integrates into an HTTP REST endpoint using FastAPI (from the docs, not run; documentation checked on 5 November 2026 at FastAPI Official Docs):

1# Production FastAPI wrapper (from the docs, not run)
2from fastapi import FastAPI, HTTPException, status
3from pydantic import BaseModel, Field
4
5app = FastAPI(title="StayWell Churn Serving API", version="1.0.0")
6
7
8class ChurnRequestSchema(BaseModel):
9    customer_id: str
10    tenure_months: int = Field(..., ge=0, le=120)
11    monthly_charges: float = Field(..., ge=0.0, le=500.0)
12    support_tickets_last_30d: int = Field(..., ge=0, le=50)
13    has_partner: bool
14    contract_type: str
15
16
17@app.post("/v1/predict", status_code=status.HTTP_200_OK)
18def predict_churn(payload: ChurnRequestSchema):
19    try:
20        response = serving_service.score(payload.dict())
21        return {
22            "customer_id": response.customer_id,
23            "churn_probability": response.churn_probability,
24            "risk_label": response.risk_label,
25            "model_version": response.model_version,
26            "latency_ms": response.execution_time_ms
27        }
28    except ValidationError as err:
29        raise HTTPException(
30            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
31            detail=str(err)
32        )

Production deployment readiness checklist

Before promoting any model serving deployment to production, the engineering and data science teams must complete and sign off on this standardized readiness checklist:

1# Model Serving Deployment Readiness Checklist
2
3## 1. Schema and Contract Validation
4- [ ] Strict input type and boundary checks are implemented at ingress.
5- [ ] Unknown or missing fields trigger HTTP 422 errors instead of silent defaults.
6- [ ] Categorical inputs are validated against an explicit enumerated set.
7- [ ] Model serialization format is platform-agnostic (ONNX, MLflow, or joblib).
8
9## 2. Latency and Scalability Guardrails
10- [ ] 95th percentile latency benchmarked under peak load (target: < 50ms for online).
11- [ ] Horizontal Pod Autoscaler (HPA) or cloud autoscale policy configured with minimum 2 replicas.
12- [ ] Health and readiness endpoints (`/healthz`, `/readyz`) respond in under 5ms without running full inference.
13- [ ] Memory limits configured to prevent Out-Of-Memory (OOM) container restarts during batch bursts.
14
15## 3. Rollout and Telemetry Infrastructure
16- [ ] Model version identifier included in every prediction response payload.
17- [ ] Prometheus metrics exported: request counts by HTTP code, latency histograms, prediction score histograms.
18- [ ] Shadow or canary deployment route configured in API gateway or service mesh.
19- [ ] Automated rollback alarm configured on 5xx error rate (> 0.1%) or latency p99 (> 150ms).
20
21## 4. Training-Serving Skew Protections
22- [ ] Feature transformations share identical code or are embedded in model pipeline graph.
23- [ ] Offline feature store table matches online low-latency key-value cache schema.
24- [ ] Sample production request payloads validated against historical training distributions.

Mistakes beginners make

1. Defaulting to online synchronous microservices for batch tasks

Engineers frequently build complex, autoscaling Kubernetes clusters to serve predictions via synchronous HTTP endpoints when downstream business consumers only need scores once a day or once a week. This introduces high operational maintenance, requires 24/7 on-call rotations, and wastes substantial cloud spend on idle container capacity.

Fix: Choose batch scoring (using Apache Spark, Airflow, or ephemeral batch jobs) whenever business processes do not require instantaneous responses to real-time user actions. Precompute scores and save them to an indexed database for instant client lookups.

2. Allowing silent failures through weak type coercion

Data science teams often pass dictionary payloads directly into Scikit-learn or PyTorch models without boundary checks. When an unexpected string, negative number, or missing feature is passed, Python either coerces the type silently or outputs inaccurate predictions without raising an error.

Fix: Implement strict defensive validation at the service ingress using strongly typed data classes or schema validation libraries. Reject malformed payloads immediately with HTTP 422 errors before the payload ever reaches model tensor code.

3. Deploying instant 100% all-at-once releases

Teams commonly update their production model deployments by updating the container image tag directly from v1 to v2 across all replicas simultaneously. If the new model version contains a subtle memory leak, an unexpected latency spike, or biased predictions on edge-case inputs, 100% of end users experience immediate degradation.

Fix: Always deploy through a shadow or canary stage. Run candidate models in shadow mode under real production traffic to verify stability, then shift live traffic gradually (10% to 50% to 100%) paired with automated rollback alarms on error rate and latency spikes.

4. Forgetting to decouple the web framework from model logic

Developers frequently intertwine web routing decorators, HTTP header parsing, and database queries directly inside the Python model inference functions. This makes it impossible to unit test model scoring logic without launching a live HTTP server, and prevents reusing the exact same model logic in batch scoring pipelines.

Fix: Architect model serving using a clean layered design. Encapsulate core model scoring and input validation inside pure, framework-agnostic Python classes. Keep web framework code (FastAPI, Flask, Triton wrappers) as a thin transport layer around the core scoring engine.

Recap and try it yourself

In this lesson, we established the core engineering patterns required to serve machine learning models reliably:

1. Pattern selection: Match your serving architecture to business latency requirements. Use batch scoring for high-throughput scheduled jobs, online REST/gRPC for sub-second user queries, and streaming pub/sub for asynchronous event-driven pipelines.
2. Defensive schema enforcement: Prevent silent model corruption by validating feature existence, strict data types, domain boundaries, and categorical enumerations before invoking the model forward pass.
3. Training-serving skew: Prevent skew by unifying feature transformation logic between training pipelines and online serving layers using serialized pipelines or shared feature stores.
4. Progressive delivery: Eliminate production release risk by validating candidate models using shadow traffic mirroring and gradual canary traffic shifting backed by automated metric watchers and rollback triggers.

Hands-on exercises

1. Implement an out-of-domain feature check: Extend the ChurnPredictionRequest.from_dict validation function in our practical code example to enforce that monthly_charges cannot be zero if tenure_months is greater than 6. Write a unit test asserting that a ValidationError is raised when an established customer has a zero monthly charge.
2. Design an automated rollback policy: Specify exact Prometheus metric thresholds for a canary deployment of StayWell's churn model. Define the PromQL queries and alerting conditions for:
   • Request error rate exceeding 0.05% over a 5-minute rolling window.
   • 99th percentile inference latency exceeding 100 milliseconds.
3. Trace a simulated shadow deployment: Take the ServingService class from this lesson. Add a shadow_score(raw_payload) method that invokes both the champion and candidate models on every request, logs both outputs to a mock dictionary, but returns only the champion response to the caller. Verify that an exception inside the candidate model does not crash the champion response.