Skip to content

11.2 Running It: Production Habits

Launch day is a moment; running is ongoing work. This lesson introduces a minimum operating plan: monitoring, recovery, controlled updates, and incident response. These practices reduce risk; they do not guarantee uninterrupted or error-free service.

What you will learn

  • Run the production trio: monitoring, backups, and the update rhythm.
  • Change live things safely — the branch-preview-merge discipline, now with stakes.
  • Handle your first incident with a five-step plan written before you needed it.

Builder principle

Production is where "probably fine" goes to get expensive. Everything you verify was cheap the day before you needed it.

The production trio

1. Something watches. From 11.1: an uptime monitor at minimum. One level up, and worth it for anything client-facing: a weekly health-check routine — and you know exactly how to build one (6.5):

# .claude/commands/health.md
Weekly production check for [project]:
1. Confirm the live URL loads; screenshot it via playwright.
2. Run scripts/check.sh against production behavior (links, form).
3. Check the uptime monitor's week: any blips?
4. Check the domain + certificate expiry dates.
5. Report: green/yellow/red per item, with evidence.

Run /health on the chosen cadence and measure how long it takes. Its value is evidence and earlier detection; it neither prevents every incident nor guarantees you learn about a failure before a user does.

2. Recovery is designed per state. Git can recover committed source. Production also has data, schemas, configuration, queues, secrets, and external side effects. Ask "if this changed or vanished tonight, what cannot I regenerate?" Give each answer a system-specific backup, migration, rollback or roll-forward path, owner, and tested recovery objective. A backup you have never restored from is a hope, not a backup.

3. Updates keep a rhythm. Live things rot quietly — dependencies age, platforms deprecate, content stales. A monthly /maintenance session (you can write this routine yourself now — the pattern is /health's): update the updatable, harness green, deploy, log it in docs/log.md. Boring by design. Boring is uptime.

Changing live things

The workflow you've run since Chapter 10 — branch, build, preview, review, merge — was training for exactly this. With production, add two disciplines:

The preview is the rehearsal. Route production changes through an isolated preview and proportionate acceptance pass when the platform supports it. A small change can still create a large effect; match the check to the affected behavior.

Deploys get receipts. After a production deploy, load the live result, run the checks most relevant to what changed, and log what shipped, when, and how it was verified. Measure the effort. The record supports later diagnosis but does not replace monitoring or broader regression coverage.

The incident plan

Someday something breaks in front of users. The plan, written now:

1. CONFIRM  — is it actually down/broken, for others or just you?
             (uptime monitor + a phone on cellular = two witnesses)
2. CONTAIN — stop harmful traffic/actions and choose the system's
             tested rollback or roll-forward. A source revert may help,
             but it does not undo schemas, data writes, config, messages,
             payments, or other external state.
3. COMMUNICATE — if anyone depends on it, one honest line now beats
             silence: "Aware of the issue, rolling back, update in 30min."
4. DIAGNOSE — calmly, on a branch, with the 6.3 method and the
             deploy receipts you kept.
5. WRITE IT DOWN — docs/incidents.md: what broke, why, what now
             prevents it. The prevention line usually becomes a
             harness check (9.3) — incidents are how harnesses grow teeth.

The load-bearing insight is step 2: contain first, then use the recovery path you tested for that system. Sometimes that is a source revert; sometimes rolling back is unsafe and a forward fix or data repair is required. The professional difference is not avoiding all incidents — it is reducing harm, communicating clearly, recovering deliberately, and leaving a check behind.

Monitoring, backups, maintenance, and incident response can define a support offer, but they also create ongoing obligations. Run the trio on your own project first; measure time, incidents, provider costs, response limits, and recovery evidence before deciding whether you can responsibly sell it.

Scaled-down SRE, deliberately — the interesting deltas are agentic: the /health and /maintenance routines are agent-run ops (headless/scheduled execution of exactly these is the natural next step — claude -p in a cron job or CI schedule, reports landing where you read them; try it on the health check this month). And incidents-become-harness-checks is your postmortem action-item pipeline with the follow-through problem solved: the action item is a test the agent writes before the postmortem doc is closed. Toil reduction, the theme of your whole field, now includes the toil of running everything you've automated.

Try it now

  1. Install the trio on your launched project: monitor + /health routine + the backup question answered (and restore tested if there's data).
  2. Write /maintenance yourself, run its first session, log it.
  3. Copy the incident plan into docs/incidents.md as the header, personalized (who gets communicated to, where rollback happens for this project).
  4. Fire drill: on an isolated preview, seed a safe failure and run the plan end to end — confirm, contain, communicate in simulation, diagnose, recover, and write up. Record the actual time and any step that required help.

Check your understanding

  • Name the trio and the one-line test of each ("a backup you've never restored from is…").
  • Why does containment beat improvised live debugging, and when is a source revert insufficient?
  • What do deploy receipts pre-gather, for which future workflow?
  • How do incidents make harnesses grow teeth?