Capstone: Pebble to Production

Suraj Kumar

Suraj Kumar

Last updated November 16, 2026·20 min read

DevOps course · lesson 24 of 24
  1. 1What Is DevOps?
  2. 2A Language and a Service to Operate
  3. 3Linux and the Terminal
  4. 4Bash Scripting for Ops
  5. 5Git and Collaboration
  6. 6Networking for DevOps
  7. 7Web Servers, Proxies and Load Balancers
  8. 8Docker for DevOps
  9. 9Docker Compose and Multi-Service Apps
  10. 10Cloud Providers: AWS, Azure and Google Cloud
  11. 11Serverless Functions and Platforms
  12. 12Terraform and Infrastructure as Code
  13. 13Ansible and Configuration Management
  14. 14Continuous Integration with GitHub Actions and GitLab CI
  15. 15Continuous Delivery and Deployment Strategies
  16. 16Secret Management
  17. 17Kubernetes Basics
  18. 18Kubernetes in Practice
  19. 19GitOps with Argo CD and Flux
  20. 20Service Mesh: Istio, Linkerd, Consul and Envoy
  21. 21Metrics and Alerting with Prometheus and Grafana
  22. 22Logs and Tracing with Loki, Elastic and OpenTelemetry
  23. 23Reliability, SLOs and Cloud Design Patterns
  24. 24Capstone: Pebble to Production
View full syllabus

After this lesson, you will be able to take one service from a commit to monitored production using every practice in this course, prove each deployment with an automated smoke test, plan and run a game day that tests your alerts, runbook and team, write a blameless postmortem from what you observed, explain honestly what your setup does not yet cover, and present the whole project in a DevOps interview. It is the last lesson of the course, and it adds very little new technology on purpose. The work now is joining the pieces and showing that they hold together.

Pebble started in A Language and a Service to Operate as one Python file with a SQLite database. Since then it has gained a systemd service, scripts, a Git workflow, a reverse proxy, a container image, PostgreSQL and a cache, cloud servers, a serverless uptime checker, Terraform, Ansible, CI and CD pipelines, secrets handling, Kubernetes manifests and a Helm chart, GitOps, a mesh experiment, metrics, logs, traces and SLOs. Each lesson built one piece in isolation. Real systems fail at the joins: an image tag that never reaches the cluster, an alert that points at the wrong target, a runbook written for last year's setup. The capstone is where you find those gaps yourself, before an interviewer or an outage finds them for you.

The whole path at a glance

The table is your checklist. Each row is a stage of delivery, the artifact you should already have and the lesson where you built it. Work through it in order and do not move on until the "done when" condition is true on your own machine or account.

StageWhat you should haveDone whenBuilt in
CodePebble 0.3.0 with its unit testspython3 -m unittest -v passesA Language and a Service to Operate, Docker Compose and Multi-Service Apps
Operations scriptsHealth check, backup and restore drillTheir test scripts pass and ShellCheck is cleanBash Scripting for Ops, Reliability, SLOs and Cloud Design Patterns
Source controlA repository with protected main and pull requestsA direct push to main is refusedGit and Collaboration
ServingNginx in front of two app replicasRequests alternate between replicas; rate limit returns 429Web Servers, Proxies and Load Balancers
ContainerA small, non-root image with a version tagBuilds, runs and scans with no critical findings you have not reviewedDocker for DevOps
Local stackCompose with proxy, app, PostgreSQL and Valkeydocker compose ps shows every service healthyDocker Compose and Multi-Service Apps
Continuous integrationTests, lint and ShellCheck on every pull requestA failing test blocks a mergeContinuous Integration with GitHub Actions and GitLab CI
DeliveryBuild once, promote the same artifactThe image digest in staging equals the one in productionContinuous Delivery and Deployment Strategies
InfrastructureTerraform with remote state; Ansible for configurationterraform plan shows no changes after applyTerraform and Infrastructure as Code, Ansible and Configuration Management
SecretsNo secret in Git, images or logsA secret scan of the repository is cleanSecret Management
KubernetesDeployment, Service, probes, resources, Helm chartkubectl get pods shows ready Pods; probes passKubernetes Basics, Kubernetes in Practice
GitOpsArgo CD syncing from a separate configuration repositoryA commit to the config repository changes the clusterGitOps with Argo CD and Flux
ObservabilityProbe, dashboard, logs and tracesGrafana shows the probe; Loki shows app logsMetrics and Alerting with Prometheus and Grafana, Logs and Tracing with Loki, Elastic and OpenTelemetry
ReliabilitySLO, burn-rate alerts, runbook, restore drillpromtool test rules passes; the drill passesReliability, SLOs and Cloud Design Patterns

Two optional stages sit outside the main path. The serverless uptime checker from Serverless Functions and Platforms gives you a second, external view of availability, and the service mesh canary from Service Mesh: Istio, Linkerd, Consul and Envoy is worth repeating if you want a progressive-delivery story. Leave both out on a small laptop; mention them as next steps instead.

Pebble's path from a commit to monitored production: a pull request runs CI, the merged commit builds one image, a config commit promotes it, Argo CD syncs it to Kubernetes, a smoke test verifies it, and probes, dashboards and burn-rate alerts watch it against the SLO.

Two repositories, one direction of change

Keep the structure from the GitOps lesson: an application repository with Pebble's code, tests, Dockerfile, CI workflow and scripts, and a configuration repository with the Helm values or manifests that describe what runs in each environment. A change flows in one direction only:

  1. A developer opens a pull request in the application repository. CI runs the tests, Ruff and ShellCheck.
  2. After review and merge, the pipeline builds one image, tags it with the version and commit, scans it and pushes it to the registry.
  3. A small commit to the configuration repository changes the image tag, or better, the digest, for staging.
  4. Argo CD notices the commit and syncs the cluster. The smoke test runs against staging.
  5. Promotion to production is another configuration commit with the same digest, after the smoke test passes and the change checklist from the reliability lesson is filled in.
  6. Probes, dashboards and burn-rate alerts watch the result. Rolling back is a revert of the configuration commit.

Nobody runs kubectl apply or helm upgrade against production by hand. If an emergency forces a manual change, the incident log records it and someone commits the same change to the configuration repository afterward, so Argo CD does not quietly undo it.

Prove every deployment with a smoke test

Unit tests prove the code works on a developer's machine. A smoke test proves a deployed copy works where users reach it: through the real proxy, with the real database and cache. It is short, fast and safe to run against production after every deployment. Pebble's smoke test checks the paths a user depends on: liveness, readiness, creating a link, the redirect, the lookup API, and that bad input is still rejected. Save it as pebble-smoke.sh in the application repository:

bash
1#!/usr/bin/env bash
2# Smoke-test a running Pebble: health, readiness, create, redirect, lookup and validation.
3set -euo pipefail
4
5base_url="${1:-}"
6[[ -n "$base_url" ]] || { echo "usage: pebble-smoke.sh BASE_URL" >&2; exit 2; }
7base_url="${base_url%/}"
8target="https://example.com/smoke-$(date -u +%Y%m%dT%H%M%SZ)"
9failures=0
10
11body=$(mktemp)
12trap 'rm -f "$body"' EXIT
13
14check() {  # check NAME EXPECTED ACTUAL
15  if [[ "$2" == "$3" ]]; then
16    echo "PASS $1"
17  else
18    echo "FAIL $1: expected $2, got $3"
19    failures=$((failures + 1))
20  fi
21}
22
23request() {  # request METHOD PATH [JSON]; prints the status code, saves the body
24  local args=(-sS -o "$body" -w '%{http_code}' --max-time 5 -X "$1")
25  if [[ -n "${3:-}" ]]; then
26    args+=(-H 'Content-Type: application/json' --data "$3")
27  fi
28  curl "${args[@]}" "$base_url$2" 2> /dev/null || true
29}
30
31field() {  # field NAME; prints one field of the saved JSON body
32  python3 -c 'import json, sys; print(json.load(sys.stdin).get(sys.argv[1], ""))' "$1" < "$body" 2> /dev/null || true
33}
34
35check "healthz" 200 "$(request GET /healthz)"
36check "readyz" 200 "$(request GET /readyz)"
37check "create link" 201 "$(request POST /api/links "{\"url\": \"$target\"}")"
38code=$(field code)
39if [[ -n "$code" ]]; then
40  redirect=$(curl -sS -o /dev/null -w '%{http_code} %{redirect_url}' --max-time 5 "$base_url/$code" 2> /dev/null || true)
41  check "redirect" "302 $target" "$redirect"
42  check "lookup status" 200 "$(request GET "/api/links/$code")"
43  check "lookup url" "$target" "$(field url)"
44else
45  check "short code returned" "a code" "nothing"
46fi
47check "reject invalid url" 400 "$(request POST /api/links '{"url": "not-a-url"}')"
48
49if (( failures > 0 )); then
50  echo "FAILED: $failures check(s) failed against $base_url"
51  exit 1
52fi
53echo "OK: all checks passed against $base_url"

A few design choices are worth noticing. Every check runs even after one fails, so a single run shows the whole picture instead of stopping at the first error. curl --max-time 5 stops a hung server from hanging the pipeline. The script prints 000 as the status when it cannot connect at all, which is what curl reports. Each run creates one link with a timestamped smoke- target, so smoke-test data is easy to recognize and exclude from reports. In a busier service you would give the smoke test its own test account or a cleanup step; for Pebble, one small row per deployment is an acceptable cost. Exit code 0 means every check passed, 1 means at least one failed and 2 means the script was called wrongly, so pipelines and people can tell the cases apart.

Start Pebble locally with PEBBLE_PORT=8080 python3 pebble.py and run the smoke test from another terminal:

bash
1./pebble-smoke.sh http://localhost:8080
2echo "exit code: $?"
text
1PASS healthz
2PASS readyz
3PASS create link
4PASS redirect
5PASS lookup status
6PASS lookup url
7PASS reject invalid url
8OK: all checks passed against http://localhost:8080
9exit code: 0

Stop Pebble and run it again:

text
1FAIL healthz: expected 200, got 000
2FAIL readyz: expected 200, got 000
3FAIL create link: expected 201, got 000
4FAIL short code returned: expected a code, got nothing
5FAIL reject invalid url: expected 400, got 000
6FAILED: 5 check(s) failed against http://localhost:8080
7exit code: 1

The test script starts a real Pebble on a free port with a temporary database, runs the smoke test against it, stops it and runs again, then points the smoke test at a plain file server that answers /healthz but is not Pebble. Save it as test-pebble-smoke.sh next to pebble.py:

bash
1#!/usr/bin/env bash
2# Run pebble-smoke.sh against a real Pebble, a stopped Pebble and a server that is not Pebble.
3set -uo pipefail
4cd "$(dirname "$0")" || exit 1
5tmp=$(mktemp -d)
6pids=()
7cleanup() { kill "${pids[@]}" 2> /dev/null; rm -r "$tmp"; }
8trap cleanup EXIT
9failures=0
10
11check() {  # check NAME EXPECTED ACTUAL
12  if [[ "$2" == "$3" ]]; then echo "ok   $1"; else echo "FAIL $1: expected '$2', got '$3'"; failures=$((failures + 1)); fi
13}
14
15free_port() { python3 -c 'import socket; s = socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])'; }
16
17wait_for() {  # wait_for URL; up to five seconds
18  for _ in $(seq 50); do curl -s -o /dev/null "$1" && return 0; sleep 0.1; done
19  return 1
20}
21
22port=$(free_port)
23PEBBLE_PORT=$port PEBBLE_DB_PATH="$tmp/pebble.db" python3 pebble.py > "$tmp/pebble.log" 2>&1 &
24pids+=($!)
25wait_for "http://127.0.0.1:$port/healthz" || { echo "Pebble did not start"; exit 1; }
26
27out=$(./pebble-smoke.sh "http://127.0.0.1:$port/")
28check "passes against a healthy Pebble" 0 $?
29check "runs all seven checks" 7 "$(grep -c '^PASS' <<< "$out")"
30
31kill "${pids[0]}"; wait "${pids[0]}" 2> /dev/null
32out=$(./pebble-smoke.sh "http://127.0.0.1:$port")
33check "fails when Pebble is stopped" 1 $?
34check "reports the failed health check" 1 "$(grep -c '^FAIL healthz' <<< "$out")"
35
36other=$(free_port)
37mkdir "$tmp/www" && echo ok > "$tmp/www/healthz"
38python3 -m http.server "$other" --bind 127.0.0.1 --directory "$tmp/www" > /dev/null 2>&1 &
39pids+=($!)
40wait_for "http://127.0.0.1:$other/healthz" || { echo "test server did not start"; exit 1; }
41out=$(./pebble-smoke.sh "http://127.0.0.1:$other")
42check "fails against a server that is not Pebble" 1 $?
43check "still reports the passing health check" 1 "$(grep -c '^PASS healthz' <<< "$out")"
44
45./pebble-smoke.sh > /dev/null 2>&1
46check "needs a base URL" 2 $?
47
48if (( failures > 0 )); then echo "$failures check(s) failed"; exit 1; fi
49echo "all checks passed"

All seven checks pass and ShellCheck reports nothing. Breaking the script on purpose, by ignoring the failure count, accepting any non-empty status, comparing against the wrong expected status, returning success on a usage error, not counting failures, or expecting the wrong redirect target, makes the tests fail every time.

Run the smoke test in three places. In CI, after the image is built, start it with Compose and run the smoke test against the proxy, which catches broken images before they reach a registry. After every deployment, run it against the environment's public URL; in a GitOps setup this can be an Argo CD PostSync hook Job or a pipeline step that waits for the sync to finish. And during incidents, run it by hand: it answers "what still works?" in two seconds.

Run a game day

A game day is a planned exercise in which the team causes a realistic failure on purpose and watches how the system and the people respond. It tests what diagrams cannot: whether the alert fires, whether it reaches someone, whether the runbook still matches the system, and whether the team knows who does what. Chaos engineering tools such as Chaos Mesh, LitmusChaos and AWS Fault Injection Service automate this kind of fault at scale; a first game day needs only a plan and a stopwatch.

Run Pebble's first game day on the local Compose stack with the observability lab and the SLO rules attached, never on a shared or production system. Write the plan before you start:

text
1Game day plan: database outage
2Hypothesis: if PostgreSQL stops, /readyz returns 503, the external probe fails,
3  PebbleErrorBudgetFastBurn fires within about ten minutes, and following the
4  runbook restores service within five minutes of the page.
5Scope: local Compose stack only. Abort if anything outside the lab is affected.
6Roles: incident commander <name>; operations <name>; observer and timekeeper <name>.
7Steps: record start time -> docker compose stop db -> observe -> follow the runbook.
8Success: alert fires; runbook leads to the fix; smoke test passes after recovery.
9Rollback: docker compose start db

Then run it:

  1. Confirm the starting state. ./pebble-smoke.sh http://localhost:8080 passes, Prometheus at http://localhost:9090/targets shows the probe UP, and the Alerts page at http://localhost:9090/alerts shows the burn-rate rules as inactive.
  2. Note the time and inject the fault from the Pebble directory with docker compose stop db.
  3. Watch without fixing. Within a few health-check intervals curl http://localhost:8080/readyz returns 503, probe_success drops to 0 in Grafana and the smoke test shows readyz and create link failing while healthz still passes, which is the liveness and readiness split working as designed.
  4. Record when PebbleErrorBudgetFastBurn moves to pending and then firing. With a 15-second scrape interval it needs about four and a half minutes of failures before the one-hour error ratio crosses 7.2% (14.4 × 0.005), plus its two-minute for delay.
  5. When it fires, the operations lead follows the runbook from the reliability lesson exactly as written, and the observer notes every step that was unclear, wrong or missing.
  6. Mitigate with docker compose start db. Confirm with the smoke test, then watch the alert return to inactive.
  7. Stop the clock and hold a short debrief while memories are fresh.

The game day often teaches more through its surprises than its successes. Perhaps the runbook says to check kubectl get pods, but the lab runs on Compose. Perhaps nobody knew where the Grafana password was. Perhaps the alert fired but nothing routed it to a person, because the lab has no Alertmanager. Each surprise is a finding, and findings belong in the postmortem.

A game day timeline: the team confirms a healthy start, stops the database, sees readiness fail and the probe drop, the fast-burn alert goes pending then fires, the runbook leads to restarting the database, the smoke test passes, the alert clears and a blameless postmortem records action items.

Write the postmortem

Use the template from the reliability lesson even though the outage was planned; practicing the format is part of the exercise. An example of how the result might read, with times relative to the start:

text
1Postmortem: Game day 1, PostgreSQL stopped              Status: reviewed
2Duration: T+0 to T+11 minutes (planned exercise, local lab only)
3Impact: in the lab, link creation and lookups failed; /healthz stayed up.
4  Error budget used (lab SLO): about 5%.
5Detection: PebbleErrorBudgetFastBurn fired at T+7. Time to detect: 7 minutes.
6Timeline: T+0 db stopped. T+1 readyz 503, probe 0. T+5 alert pending.
7  T+7 alert firing. T+8 runbook step 4 identifies the database. T+9 db started.
8  T+10 smoke test passes. T+11 alert inactive.
9Contributing causes: single PostgreSQL instance with no standby; the runbook
10  checked Kubernetes before Compose; no route from alerts to a person in the lab.
11What went well: readiness and liveness split; alert fired as the unit test predicted.
12What was hard: the runbook's first command did not apply to the lab.
13Action items:
14  Add a Compose branch to runbook step 4 | <owner> | P2 | <date> | <ticket>
15  Add Alertmanager with an email receiver to the lab | <owner> | P2 | <date> | <ticket>
16  Write a DR note on adding a PostgreSQL standby | <owner> | P3 | <date> | <ticket>

Your own timeline will differ; record what you actually saw. Then do the action items, and run the same game day again. A second run that is faster and calmer is the evidence that the loop works.

Make the project presentable

A capstone that only runs on your laptop is hard to show anyone. Turn it into a portfolio project:

  • A README.md in the application repository with a one-paragraph description, an architecture diagram, how to run it locally in three commands, and links to the configuration repository and the pipeline runs.
  • The pipeline status badge and a green run history, so a reader can see CI working without cloning anything.
  • A docs/ folder with the SLO sheet, runbook, disaster recovery plan and the game-day postmortem.
  • A short "decisions" section explaining why you chose each tool and what you would change at a larger scale, such as PostgreSQL with a standby, a managed Kubernetes service or Alertmanager routing to an on-call rotation.
  • Screenshots of the Grafana dashboard during the game day, the Argo CD application tree and the failing-then-passing smoke test.

Keep secrets, account IDs and personal data out of everything you publish, including screenshots and Terraform state. The secret-scanning step from the secrets lesson should run on this repository too.

What this capstone does not cover

Be honest about limits; interviewers trust candidates who know where their system stops. This project deliberately leaves out:

  • High availability of data. Pebble runs one PostgreSQL instance. Production would use a managed database with a standby or replicas, automated backups and point-in-time recovery, and the restore drill would restore into that.
  • Real multi-zone or multi-region resilience. A kind cluster is several containers on one machine. The ideas carry over to a managed cluster across availability zones, but the failure modes and costs are different.
  • Request-level metrics. Pebble has no /metrics endpoint, so its SLI comes from probes. Adding Prometheus client metrics for request counts and latency would allow a request-based SLO and the four golden signals.
  • Alert routing and on-call. The lab shows alerts in Prometheus. A real team routes them through Alertmanager or a paging service to a rotation with escalation.
  • Supply chain security. Image signing, software bills of materials, provenance attestations and admission policies that only allow signed images are the next layer after scanning.
  • Cost management. The course mentions budgets and cleanup, but a production service needs cost allocation tags, alerts on spend and regular reviews.
  • Compliance and data protection. Audit trails, retention rules and privacy requirements depend on the business and the country.

None of these gaps is a failure. Each is a sensible next step, and naming them shows you understand the difference between a learning environment and production.

Where to go next

Choose depth in one direction rather than breadth in all of them:

  • Platform engineering: build an internal developer platform that gives other teams Pebble-style pipelines and environments through templates and self-service.
  • Kubernetes depth: operators and custom resources, network policies, autoscaling on custom metrics, and the Cloud Native Computing Foundation's Kubernetes certifications if a credential matters for your job search.
  • One cloud in depth: pick the provider your target employers use and learn its networking, identity, managed databases and cost tools well.
  • Security: the Cyber Security course on this site goes deeper into threats, defenses and incident response, which pairs naturally with DevSecOps work.
  • Reliability: read the free Google SRE book and workbook in full, and practice writing SLOs for services you use every day.

Interview preparation checklist

DevOps interviews usually mix fundamentals, scenario questions and a discussion of something you built. Use this checklist in the weeks before:

  1. Explain DevOps in two minutes: culture, automation, measurement and sharing, the DORA metrics, and how SRE and platform engineering relate. Practice out loud.
  2. Linux and networking fundamentals: permissions, processes, systemd, logs, DNS resolution, TLS, what happens when you type a URL, and how you would debug "the site is slow".
  3. Write a script live: a short Bash or Python script that reads a log and counts errors, with exit codes and quoting done right.
  4. Containers and Kubernetes: image layers, why containers run as non-root, Deployments versus StatefulSets, readiness versus liveness probes, requests versus limits, and how a Service finds Pods.
  5. CI/CD and GitOps: build once and promote, deployment strategies and when to use each, rollback versus roll forward, and why GitOps uses a separate configuration repository.
  6. Infrastructure as code: what Terraform state is and why it needs locking, how to handle drift, and when to use configuration management instead.
  7. Observability and reliability: metrics, logs and traces with an example of each; SLI, SLO and error budget in your own words; burn-rate alerting; a blameless postmortem.
  8. Security basics: where secrets leak, least privilege, workload identity, image scanning and secret scanning.
  9. Tell the capstone story using situation, task, action and result: what you built, one hard problem and how you solved it, what the game day revealed, what you would change next. Keep it under three minutes and have the repository open.
  10. Prepare questions for them: how they deploy, how often, how they measure reliability, what their on-call looks like and what happened in their last incident.

Expect at least one scenario question, such as "a deployment just went out and error rates doubled; what do you do?" A strong answer follows the order from this course: mitigate first by rolling back, communicate, confirm recovery with data, then investigate and write it up.

Mistakes beginners make

  • Building every piece separately and never testing the joins between them.
  • Deploying to production by hand "just this once" and letting the configuration repository drift.
  • Treating a passing unit test suite as proof that a deployment works, without a smoke test.
  • Running a first game day on a shared or production environment.
  • Fixing the fault during a game day before the alert and runbook have been tested.
  • Writing a postmortem for a game day that hides the awkward parts.
  • Publishing a portfolio repository with secrets, account IDs or Terraform state in it.
  • Claiming a capstone is "production-grade" in an interview instead of explaining its limits.
  • Learning a little of every tool on a job ad instead of explaining one working system well.

Recap and try it yourself

The capstone joins the course into one system: code and tests, scripts, Git, a proxy, a container, Compose, CI, build-once delivery, Terraform and Ansible, managed secrets, Kubernetes with Helm, GitOps with Argo CD, metrics, logs and traces, and SLOs with burn-rate alerts. Changes flow one way, from pull request to image to configuration commit to cluster, and rollback is a revert. A smoke test proves each deployment where users reach it. A game day tests the alerts, runbook and roles under controlled conditions, and its blameless postmortem turns surprises into action items. Knowing the limits of the setup, and the next steps beyond it, is part of doing DevOps well.

Try it yourself:

  1. Work through the path table and record, for each row, the command or link that proves it is done.
  2. Add pebble-smoke.sh to your CI pipeline so it runs against the Compose stack after the image builds, and make a failing smoke test block the merge.
  3. Run the database-outage game day, write the postmortem with your real timeline, complete one action item and run it again.
  4. Design a second game day: stop the Valkey cache, or delete one app Pod in kind. Write the hypothesis first, then check it.
  5. Write your portfolio README.md and a three-minute capstone story, and practice it out loud with someone who will ask "why?" at least twice.

Sources checked on 16 November 2026: Google SRE workbook chapters on alerting on SLOs and postmortem culture; the Google SRE book chapter on managing incidents; AWS Well-Architected guidance on game days; Argo CD resource hooks; curl's --write-out variables; the Chaos Mesh, LitmusChaos and AWS Fault Injection Service project pages; and the roadmap.sh DevOps roadmap. Pebble, its game day and its postmortem are fictional.

Related lessons in other courses