Phase 3: release script, container limits, rehearsed rollback

Adds the deployment mechanism for the portfolio site. The site itself is not deployed yet: Phase 2 has not produced a binary. - deploy/deploy.sh: build + test (fail fast) -> releases/<ts> -> atomic `current` symlink flip -> docker compose up -d -> /healthz gate -> flip back to `previous` and exit non-zero on failure. `--rollback` restores the previous release in one command. Auto-detects the Go toolchain in /nix/store when `go` is not on PATH. - deploy/docker-compose.yml: 127.0.0.1:8096, mem_limit == memswap_limit == 120m, capped json-file logs (10m x 3), restart unless-stopped. - deploy/Dockerfile: runtime image only (alpine + prebuilt payload), so a release needs no toolchain, no network and no package install. - deploy/rehearse.sh: repeatable proof (21 assertions, exit 0) of deploy, build fail-fast, health-gate rollback, --rollback and N3 restart. - deploy/README.md + docs/phase-3-evidence.md: how it works, the build contract Phase 2 must satisfy, the Caddy block for Sam, and the open bind-address question (loopback-only vs reachable from .35). Satisfies DoD items 1, 3 (container), 4, 8. Items 2, 5, 6, 7 need the Phase 2 binary and Sam's Caddy change; recorded as deferred with owners, not claimed. Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-26 17:31:13 +11:00
parent b267e55f8c
commit 8b3209fab5
6 changed files with 880 additions and 0 deletions
--- a/deploy/Dockerfile
+++ b/deploy/Dockerfile
@@ -0,0 +1,28 @@
+# resume — runtime image (Phase 3 §3.2).
+#
+# Built by deploy/deploy.sh from a release directory, with the prebuilt static
+# Go binary already in place. No Go toolchain, no network and no package install
+# at build time: the image is `alpine` + the release payload, nothing else.
+#
+# The binary must be statically linked (deploy.sh's fallback build sets
+# CGO_ENABLED=0). If the app ever needs cgo, change the base image to a glibc
+# one (e.g. golang:1.25-bookworm) in git — do not patch it on the host.
+
+FROM alpine:3.21
+
+ARG APP_VERSION=dev
+LABEL org.opencontainers.image.title="resume" \
+      org.opencontainers.image.version="${APP_VERSION}" \
+      org.opencontainers.image.description="Sam Rolfe — portfolio site (Mode A static)"
+
+WORKDIR /srv
+
+# bin/server plus whatever the build emitted (dist/ for rendered pages/assets).
+COPY . /srv/
+
+ENV PORT=8096 \
+    APP_VERSION=${APP_VERSION}
+
+EXPOSE 8096
+
+ENTRYPOINT ["/srv/bin/server"]
--- a/deploy/README.md
+++ b/deploy/README.md
@@ -0,0 +1,93 @@
+# Deployment (Phase 3)
+
+One script, one release directory, one atomic symlink. Deploy is a script, not a
+model decision (`AGENTS.md` §4.6), and nobody hand-edits files on the runtime
+host: change these files in git and deploy again.
+
+| File | Purpose |
+|---|---|
+| `deploy.sh` | build + test → release → flip → `up -d` → health gate → rollback on failure |
+| `deploy.sh --rollback` | restore the previous release in one command |
+| `docker-compose.yml` | runtime container: `127.0.0.1:8096`, `mem_limit == memswap_limit == 120m`, capped logs |
+| `Dockerfile` | runtime image only — alpine + the prebuilt release payload, no toolchain, no network |
+| `rehearse.sh` | repeatable proof that deploy, fail-fast, health-gate rollback and `--rollback` all work |
+
+## Layout on `.13`
+
+```
+/home/sam/deployment/resume/
+├── releases/<UTC timestamp>/     immutable: bin/server, dist/, compose, Dockerfile, manifest.txt, .env
+├── current  -> releases/<ts>     live release; swapped with ln + mv -T (atomic rename)
+├── previous -> releases/<ts>     the release that was live before the last flip
+└── deploy.log                    append-only record of every attempt, including failures
+```
+
+## What `deploy.sh` expects from the app
+
+1. `make build` if a `Makefile` with a `build` target exists, otherwise
+   `go build ./cmd/server`. Either way the result must be one server binary at
+   `bin/server` or `dist/server`; anything else the binary serves from disk goes
+   in `dist/` and is copied into the release.
+2. `make test` (or `go test ./...`) — runs **before** anything is copied or
+   flipped, so a broken build never reaches the release directory.
+3. `GET /healthz` → `200`. If the body is JSON with a `static` field it must be
+   `true`. AI readiness is deliberately **not** part of the gate: Mode A must
+   work with every AI service stopped (acceptance test A1).
+4. Bind `$PORT` (default `8096`) on `0.0.0.0` **inside the container**. The
+   container's own loopback is not reachable through a published port; exposure
+   is controlled by the host side of `ports:` in `docker-compose.yml`.
+
+## Deploying
+
+```bash
+cd /home/sam/paseo/projects/resume_ai_system_paperclip_build
+./deploy/deploy.sh                 # build, test, release, flip, health gate
+./deploy/deploy.sh --rollback      # one command back to the previous release
+tail -40 /home/sam/deployment/resume/deploy.log
+```
+
+Exit code is the answer: `0` means the new release answered `/healthz`.
+Non-zero means either (a) build/test failed and **nothing changed**, or (b) the
+health gate failed, `current` was flipped back to `previous`, the previous
+release was restarted, and the log says which.
+
+Flags worth knowing: `--source DIR` (build from another checkout — this is how
+`rehearse.sh` drives the script), `--root DIR`, `--port N`,
+`--health-timeout N`.
+
+The Go toolchain is auto-detected, including `/nix/store/*-go-1.*/bin/go`, so
+the script works on a host where `go` is not on `PATH`.
+
+## Rehearsing the rollback
+
+```bash
+./deploy/rehearse.sh        # 21 assertions, exit 0 when every one holds
+```
+
+It builds three throwaway services in a scratch directory, runs the real script
+against the real release root and the real port, and asserts:
+
+1. a good release deploys, answers `/healthz`, publishes on `127.0.0.1` only,
+   and has `memswap_limit == mem_limit`;
+2. a build failure exits non-zero, leaves `current` untouched and keeps serving;
+3. a release whose `/healthz` is `500` is rolled back by the health gate, exits
+   non-zero, and the previous release serves again;
+4. `deploy.sh --rollback` flips back to the previous release and exits 0;
+5. `docker compose down && up` (acceptance test N3) restores the service.
+
+It removes the release root, container and images it created. `KEEP_REHEARSAL=1`
+keeps them for inspection.
+
+## Container limits — not negotiable
+
+`mem_limit` and `memswap_limit` are both `120m`, so the container cannot eat
+host swap on a host that is already swapping. Measured RSS for a Go static
+server is single-digit MiB; `deploy.sh` prints the real figure after every
+deploy so drift shows up in the log.
+
+## Caddy
+
+`docs/phase-3-deploy.md` §3.3 holds the block for `.35`. It is additive, applied
+with `caddy reload` (never a restart), and **Sam applies it** — see
+`docs/phase-3-evidence.md` for the open bind-address question that decides
+whether that block points at `192.168.20.13:8096` or at `.13`'s Caddy.
--- a/deploy/deploy.sh
+++ b/deploy/deploy.sh
@@ -0,0 +1,303 @@
+#!/usr/bin/env bash
+# resume — release script (Phase 3).
+#
+# Deploy is a script, not a model decision (AGENTS.md §4.6).
+# No one hand-edits files on the production host: change this script in git.
+#
+#   ./deploy/deploy.sh                 build, test, release, flip, health-gate
+#   ./deploy/deploy.sh --rollback      restore the previous release, one command
+#
+# Layout on the runtime host (.13):
+#
+#   /home/sam/deployment/resume/
+#   ├── releases/<UTC timestamp>/      immutable release: bin/, dist/, compose, manifest
+#   ├── current -> releases/<ts>       the live release (atomic symlink flip)
+#   ├── previous -> releases/<ts>      the release that was live before the last flip
+#   └── deploy.log                     append-only record of every deploy attempt
+#
+# A release is only "current" after /healthz answers 200 on 127.0.0.1:$PORT.
+# If the health gate fails, `current` is flipped back to `previous`, the previous
+# release is restarted, and this script exits non-zero.
+
+set -Eeuo pipefail
+
+APP_NAME="resume"
+DEFAULT_PORT=8096
+DEFAULT_ROOT="/home/sam/deployment/resume"
+
+SRC_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+ROOT="${DEPLOY_ROOT:-$DEFAULT_ROOT}"
+PORT="${APP_PORT:-$DEFAULT_PORT}"
+DO_ROLLBACK=0
+HEALTH_TIMEOUT="${HEALTH_TIMEOUT:-40}"
+
+usage() {
+  cat <<'EOF'
+Usage: deploy.sh [options]
+
+  --rollback            restore the previous release and exit
+  --source DIR          build from DIR instead of the repo this script lives in
+  --root DIR            release root (default /home/sam/deployment/resume)
+  --port N              app port (default 8096)
+  --health-timeout N    seconds to wait for /healthz (default 40)
+  -h, --help            this text
+
+Environment:
+  GO            path to the go toolchain (auto-detected, incl. /nix/store)
+  DEPLOY_ROOT   same as --root
+  APP_PORT      same as --port
+EOF
+}
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --rollback)       DO_ROLLBACK=1; shift ;;
+    --source)         SRC_DIR="$(cd "$2" && pwd)"; shift 2 ;;
+    --root)           ROOT="$2"; shift 2 ;;
+    --port)           PORT="$2"; shift 2 ;;
+    --health-timeout) HEALTH_TIMEOUT="$2"; shift 2 ;;
+    -h|--help)        usage; exit 0 ;;
+    *) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;;
+  esac
+done
+
+RELEASES="$ROOT/releases"
+CURRENT="$ROOT/current"
+PREVIOUS="$ROOT/previous"
+LOG="$ROOT/deploy.log"
+
+mkdir -p "$RELEASES"
+
+log() { printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" | tee -a "$LOG" >&2; }
+die() { log "FAIL: $*"; exit 1; }
+
+# ---------------------------------------------------------------- toolchain
+find_go() {
+  if [[ -n "${GO:-}" && -x "${GO}" ]]; then echo "$GO"; return; fi
+  if command -v go >/dev/null 2>&1; then command -v go; return; fi
+  # NixOS: the toolchain lives in the store even when it is not on PATH.
+  local cand
+  cand=$(ls -d /nix/store/*-go-1.*/bin/go 2>/dev/null | grep -v '\.drv' | sort -V | tail -1)
+  if [[ -n "$cand" && -x "$cand" ]]; then echo "$cand"; return; fi
+  return 1
+}
+
+compose() {
+  # $1 = release directory that holds docker-compose.yml + .env
+  local dir="$1"; shift
+  docker compose -p "$APP_NAME" --project-directory "$dir" -f "$dir/docker-compose.yml" "$@"
+}
+
+# ---------------------------------------------------------------- health gate
+check_health() {
+  # 200 on /healthz, and if the body is JSON with a "static" field it must be true.
+  # AI readiness is deliberately NOT part of this gate: Mode A must work with
+  # every AI service stopped (acceptance test A1).
+  local url="http://127.0.0.1:${PORT}/healthz" body code
+  body=$(curl -sS --max-time 3 -w '\n%{http_code}' "$url" 2>/dev/null || true)
+  code=$(printf '%s' "$body" | tail -1)
+  body=$(printf '%s' "$body" | sed '$d')
+  if [[ "$code" != "200" ]]; then
+    log "health: $url -> HTTP ${code:-none}"
+    return 1
+  fi
+  if printf '%s' "$body" | head -c 1 | grep -q '{' && command -v jq >/dev/null 2>&1; then
+    local static_ok
+    static_ok=$(printf '%s' "$body" | jq -r 'if has("static") then (.static|tostring) else "absent" end' 2>/dev/null || echo absent)
+    if [[ "$static_ok" == "false" ]]; then
+      log "health: $url -> 200 but static readiness is false: $body"
+      return 1
+    fi
+  fi
+  log "health: $url -> 200"
+  return 0
+}
+
+wait_healthy() {
+  local i
+  for ((i = 1; i <= HEALTH_TIMEOUT; i++)); do
+    if check_health; then return 0; fi
+    sleep 1
+  done
+  return 1
+}
+
+container_state() {
+  docker inspect -f 'name={{.Name}} status={{.State.Status}} mem={{.HostConfig.Memory}} memswap={{.HostConfig.MemorySwap}} ports={{json .HostConfig.PortBindings}}' \
+    "${APP_NAME}-app" 2>/dev/null || echo "container ${APP_NAME}-app: absent"
+}
+
+# ---------------------------------------------------------------- build + test
+discover_server() {
+  # Contract (deploy/README.md): the app build produces one server binary at
+  # bin/server or dist/server. `make build` may also emit dist/ for static assets.
+  local d
+  for d in "$1/bin/server" "$1/dist/server" "$1/server"; do
+    if [[ -f "$d" && -x "$d" ]]; then echo "$d"; return 0; fi
+  done
+  return 1
+}
+
+build_and_test() {
+  local src="$1" out="$2"
+  local gobin=""
+  log "build: source $src"
+  mkdir -p "$out/bin"
+  cd "$src"
+
+  gobin=$(find_go || true)
+  if [[ -z "$gobin" ]]; then
+    log "build: no go toolchain found (set GO=/path/to/go)"
+  else
+    export PATH="$(dirname "$gobin"):$PATH"
+    log "build: go $("$gobin" version)"
+  fi
+
+  if [[ -f Makefile ]] && grep -qE '^build:' Makefile; then
+    log "build: make build"
+    make build
+  else
+    [[ -n "$gobin" ]] || die "no Makefile 'build' target and no go toolchain"
+    log "build: go build ./cmd/server"
+    CGO_ENABLED=0 "$gobin" build -trimpath -buildvcs=false -o "$out/bin/server" ./cmd/server
+  fi
+
+  # Tests fail fast, before anything is copied or flipped.
+  if [[ -f Makefile ]] && grep -qE '^test:' Makefile; then
+    log "test: make test"
+    make test
+  elif [[ -n "$gobin" ]]; then
+    log "test: go test ./..."
+    "$gobin" test ./...
+  fi
+
+  # Collect the binary, wherever the app's build put it.
+  local bin
+  bin=$(discover_server "$src" || true)
+  if [[ -z "$bin" ]]; then
+    bin=$(discover_server "$out" || true)
+  fi
+  [[ -n "$bin" ]] || die "no server binary found (looked for bin/server, dist/server, server)"
+  if [[ "$(readlink -f "$bin")" != "$(readlink -f "$out/bin/server")" ]]; then
+    cp -f "$bin" "$out/bin/server"
+  fi
+  chmod +x "$out/bin/server"
+  log "build: binary $(stat -c %s "$out/bin/server") bytes, sha256 $(sha256sum "$out/bin/server" | cut -c1-16)"
+
+  # Static assets, if the build emits them.
+  if [[ -d "$src/dist" ]]; then
+    cp -a "$src/dist/." "$out/dist/" 2>/dev/null || { mkdir -p "$out/dist"; cp -a "$src/dist/." "$out/dist/"; }
+    log "build: copied dist/ ($(find "$out/dist" -type f | wc -l) files)"
+  fi
+}
+
+# ---------------------------------------------------------------- deploy
+do_deploy() {
+  local src="$1"
+  local ts sha branch release prev_target
+  ts=$(date -u +%Y%m%dT%H%M%SZ)
+  sha=$(git -C "$src" rev-parse --short HEAD 2>/dev/null || echo unknown)
+  branch=$(git -C "$src" rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown)
+  release="$RELEASES/$ts"
+
+  log "deploy: $APP_NAME $ts (branch $branch, $sha) -> $release"
+  mkdir -p "$release"
+
+  # 1. Build + test. Nothing is touched on failure.
+  build_and_test "$src" "$release"
+
+  # 2. Release payload: binary, compose file, runtime image, manifest.
+  cp -f "$src/deploy/docker-compose.yml" "$release/docker-compose.yml"
+  cp -f "$src/deploy/Dockerfile" "$release/Dockerfile"
+  {
+    echo "app=$APP_NAME"
+    echo "release=$ts"
+    echo "git_sha=$(git -C "$src" rev-parse HEAD 2>/dev/null || echo unknown)"
+    echo "git_branch=$branch"
+    echo "built_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+    echo "built_on=$(hostname)"
+    echo "binary_sha256=$(sha256sum "$release/bin/server" | cut -d' ' -f1)"
+  } > "$release/manifest.txt"
+  printf 'APP_VERSION=%s\nAPP_PORT=%s\n' "$ts" "$PORT" > "$release/.env"
+
+  # 3. Runtime image, tagged with this release.
+  log "deploy: docker build resume/app:$ts"
+  docker build --quiet -t "resume/app:$ts" \
+    --build-arg "APP_VERSION=$ts" -f "$release/Dockerfile" "$release" >>"$LOG" 2>&1 \
+    || die "image build failed"
+
+  # 4. Remember what is live now, then flip atomically.
+  prev_target=""
+  if [[ -L "$CURRENT" ]]; then prev_target=$(readlink -f "$CURRENT"); fi
+  ln -sfn "$release" "$CURRENT.tmp" && mv -T "$CURRENT.tmp" "$CURRENT" || die "symlink flip failed"
+  if [[ -n "$prev_target" && "$prev_target" != "$(readlink -f "$release")" ]]; then
+    ln -sfn "$prev_target" "$PREVIOUS.tmp" && mv -T "$PREVIOUS.tmp" "$PREVIOUS"
+    log "deploy: previous -> $prev_target"
+  fi
+  log "deploy: current -> $(readlink -f "$CURRENT")"
+
+  # 5. Bring up, then gate on health. Anything less than 200 means roll back.
+  compose "$release" up -d --remove-orphans >>"$LOG" 2>&1 || log "deploy: compose up returned non-zero"
+  if ! wait_healthy; then
+    log "deploy: HEALTH GATE FAILED for $ts"
+    rollback_internal "$ts"
+    exit 1
+  fi
+
+  log "deploy: OK $ts"
+  container_state | tee -a "$LOG" >&2
+  docker stats --no-stream --format 'rss={{.MemUsage}} pct={{.MemPerc}}' "${APP_NAME}-app" 2>/dev/null | tee -a "$LOG" >&2 || true
+}
+
+rollback_internal() {
+  # Used by a failed deploy: restore the previous release, restart it, verify.
+  local failed="$1" target
+  if [[ ! -L "$PREVIOUS" ]]; then
+    log "rollback: no previous release to restore; stopping ${APP_NAME}-app"
+    compose "$CURRENT" down >>"$LOG" 2>&1 || true
+    return 1
+  fi
+  target=$(readlink -f "$PREVIOUS")
+  log "rollback: $failed failed -> restoring $target"
+  ln -sfn "$target" "$CURRENT.tmp" && mv -T "$CURRENT.tmp" "$CURRENT"
+  compose "$target" up -d --remove-orphans >>"$LOG" 2>&1 || true
+  if wait_healthy; then
+    log "rollback: previous release healthy ($(readlink -f "$CURRENT"))"
+  else
+    log "rollback: previous release ALSO unhealthy — manual attention required"
+    return 1
+  fi
+}
+
+do_rollback() {
+  local target
+  if [[ ! -L "$PREVIOUS" ]]; then die "no previous release at $PREVIOUS — nothing to roll back to"; fi
+  target=$(readlink -f "$PREVIOUS")
+  [[ -d "$target" ]] || die "previous release target missing: $target"
+  log "rollback: current=$( [[ -L "$CURRENT" ]] && readlink -f "$CURRENT" || echo none) -> $target"
+
+  # Keep the release we are leaving, so the rollback is itself reversible.
+  local leaving=""
+  if [[ -L "$CURRENT" ]]; then leaving=$(readlink -f "$CURRENT"); fi
+  ln -sfn "$target" "$CURRENT.tmp" && mv -T "$CURRENT.tmp" "$CURRENT" || die "symlink flip failed"
+  if [[ -n "$leaving" && "$leaving" != "$target" ]]; then
+    ln -sfn "$leaving" "$PREVIOUS.tmp" && mv -T "$PREVIOUS.tmp" "$PREVIOUS"
+  fi
+
+  compose "$target" up -d --remove-orphans >>"$LOG" 2>&1 || log "rollback: compose up returned non-zero"
+  if wait_healthy; then
+    log "rollback: OK, current -> $target"
+    container_state | tee -a "$LOG" >&2
+    exit 0
+  fi
+  log "rollback: FAILED, $target is not healthy"
+  exit 1
+}
+
+mkdir -p "$ROOT"
+log "=== ${APP_NAME} deploy.sh start (pid $$) ==="
+if (( DO_ROLLBACK == 1 )); then
+  do_rollback
+else
+  do_deploy "$SRC_DIR"
+fi
--- a/deploy/docker-compose.yml
+++ b/deploy/docker-compose.yml
@@ -0,0 +1,41 @@
+# resume — runtime container (Phase 3 §3.2).
+#
+# Copied into each release directory by deploy/deploy.sh; the release's .env
+# supplies APP_VERSION (the image tag). Never edited on the host — change it
+# here, in git, and deploy again.
+#
+# Limits that are not negotiable (AGENTS.md §4.3):
+#   mem_limit == memswap_limit == 120m   -> the container cannot touch host swap
+#   ports 127.0.0.1:$PORT:8096           -> Caddy on .35 is the only public listener
+#
+# .13 has every default Docker address pool subnetted out (many projects), so a
+# single-container service uses the host bridge and publishes its port rather
+# than creating a project network. Same pattern as compass-teams-nhs.
+
+name: resume
+
+services:
+  app:
+    image: resume/app:${APP_VERSION}
+    container_name: resume-app
+    restart: unless-stopped
+    network_mode: bridge
+    ports:
+      - "127.0.0.1:${APP_PORT:-8096}:8096"
+    environment:
+      PORT: "8096"
+      APP_VERSION: "${APP_VERSION}"
+      # Mode B values arrive in Phase 5 from ~/.config/environment.d/ — never here.
+    mem_limit: 120m
+    memswap_limit: 120m
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+    healthcheck:
+      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8096/healthz"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 5s
--- a/deploy/rehearse.sh
+++ b/deploy/rehearse.sh
@@ -0,0 +1,184 @@
+#!/usr/bin/env bash
+# resume — rollback rehearsal (Phase 3 DoD: "Rollback rehearsed once and documented").
+#
+# Builds three throwaway Go services in a scratch directory and runs the real
+# deploy mechanism against the real release root and the real port:
+#
+#   1. good       -> deploys, health-gated, serving
+#   2. broken     -> does not compile: deploy must fail fast, current must not move
+#   3. unhealthy  -> builds and runs, but /healthz is 500: the health gate must
+#                    roll back to the previous release and exit non-zero
+#   4. good again -> then `deploy.sh --rollback` must restore the earlier release
+#
+# It leaves nothing running and removes the release root it created, unless
+# KEEP_REHEARSAL=1. Nothing outside $DEPLOY_ROOT, the scratch dir and this
+# service's own container/image is touched.
+#
+#   ./deploy/rehearse.sh
+#
+# Exit 0 means every assertion held; the transcript is the evidence.
+
+set -Eeuo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+DEPLOY="$HERE/deploy.sh"
+ROOT="${DEPLOY_ROOT:-/home/sam/deployment/resume}"
+PORT="${APP_PORT:-8096}"
+SCRATCH="${REHEARSAL_DIR:-$(mktemp -d -t resume-rehearsal-XXXXXX)}"
+KEEP="${KEEP_REHEARSAL:-0}"
+
+pass=0; fail=0
+ok()   { printf '  PASS  %s\n' "$*"; pass=$((pass + 1)); }
+no()   { printf '  FAIL  %s\n' "$*"; fail=$((fail + 1)); }
+hdr()  { printf '\n== %s\n' "$*"; }
+
+cleanup() {
+  if docker ps -a --format '{{.Names}}' | grep -qx 'resume-app'; then
+    docker rm -f resume-app >/dev/null 2>&1 || true
+  fi
+  docker images --format '{{.Repository}}:{{.Tag}}' | grep -q '^resume/app:' \
+    && docker images --format '{{.Repository}}:{{.Tag}}' | grep '^resume/app:' \
+       | xargs -r docker rmi -f >/dev/null 2>&1 || true
+  if [[ "$KEEP" == "1" ]]; then
+    printf '\nkept: release root %s, scratch %s, images resume/app:*\n' "$ROOT" "$SCRATCH"
+  else
+    rm -rf "$ROOT"
+    [[ "$SCRATCH" == /tmp/resume-rehearsal-* || -n "${REHEARSAL_DIR:-}" ]] && rm -rf "$SCRATCH"
+    printf '\ncleaned: release root and rehearsal sources removed\n'
+  fi
+}
+trap cleanup EXIT
+
+# --------------------------------------------------------------- fixtures
+make_app() { # $1 dir, $2 mode: good|broken|unhealthy
+  local dir="$1" mode="$2"
+  mkdir -p "$dir/cmd/server" "$dir/deploy"
+  cp -f "$HERE/docker-compose.yml" "$HERE/Dockerfile" "$dir/deploy/"
+  printf 'module rehearsal\n\ngo 1.25\n' > "$dir/go.mod"
+  cat > "$dir/cmd/server/main.go" <<'GO'
+package main
+
+import (
+	"fmt"
+	"log"
+	"net/http"
+	"os"
+	"time"
+)
+
+func main() {
+	port := os.Getenv("PORT")
+	if port == "" {
+		port = "8096"
+	}
+	bind := os.Getenv("BIND")
+	if bind == "" {
+		bind = "0.0.0.0"
+	}
+	mux := http.NewServeMux()
+	mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
+		BROKEN
+		w.Header().Set("Content-Type", "application/json")
+		fmt.Fprintf(w, `{"status":"ok","static":true,"ai":"disabled","version":%q,"time":%q}`,
+			os.Getenv("APP_VERSION"), time.Now().UTC().Format(time.RFC3339))
+	})
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/html; charset=utf-8")
+		fmt.Fprintf(w, "<!doctype html><title>rehearsal</title><p>version %s</p>", os.Getenv("APP_VERSION"))
+	})
+	srv := &http.Server{Addr: bind + ":" + port, Handler: mux, ReadHeaderTimeout: 5 * time.Second}
+	log.Printf("listening on %s:%s", bind, port)
+	if err := srv.ListenAndServe(); err != nil {
+		log.Fatal(err)
+	}
+}
+GO
+  local patch=''
+  case "$mode" in
+    good)      patch='' ;;
+    unhealthy) patch='w.WriteHeader(http.StatusInternalServerError)' ;;
+    broken)    patch='' ;;
+  esac
+  python3 - "$dir/cmd/server/main.go" "$patch" <<'PY'
+import sys
+p, patch = sys.argv[1], sys.argv[2]
+s = open(p).read().replace("BROKEN", patch, 1)
+open(p, "w").write(s)
+PY
+  if [[ "$mode" == "broken" ]]; then
+    sed -i 's/^func main() {/func main( {/' "$dir/cmd/server/main.go"
+  fi
+}
+
+health() { curl -s -m 3 -o /dev/null -w '%{http_code}' "http://127.0.0.1:$PORT/healthz" || true; }
+serving_version() { curl -s -m 3 "http://127.0.0.1:$PORT/healthz" | sed -n 's/.*"version":"\([^"]*\)".*/\1/p'; }
+current_target() { [[ -L "$ROOT/current" ]] && readlink -f "$ROOT/current" || echo none; }
+
+# --------------------------------------------------------------- 0. preconditions
+hdr "0. preconditions"
+if ss -ltn 2>/dev/null | awk '{print $4}' | grep -q ":$PORT\$"; then
+  no "port $PORT already in use — nothing started"
+  exit 1
+fi
+ok "port $PORT free"
+rm -rf "$ROOT"
+make_app "$SCRATCH/good" good
+make_app "$SCRATCH/broken" broken
+make_app "$SCRATCH/unhealthy" unhealthy
+ok "three throwaway sources built in $SCRATCH"
+
+# --------------------------------------------------------------- 1. good deploy
+hdr "1. good release deploys and health-gates"
+if "$DEPLOY" --source "$SCRATCH/good" > "$SCRATCH/1-deploy.log" 2>&1; then
+  ok "deploy.sh exit 0"
+else
+  no "deploy.sh exit non-zero"; tail -20 "$SCRATCH/1-deploy.log"; exit 1
+fi
+[[ "$(health)" == "200" ]] && ok "/healthz 200 on 127.0.0.1:$PORT" || no "/healthz not 200"
+v1=$(serving_version); [[ -n "$v1" ]] && ok "serving release $v1" || no "no version reported"
+ss -ltn | grep -q "127.0.0.1:$PORT" && ok "container publishes on 127.0.0.1 only" || no "unexpected bind"
+mem=$(docker inspect -f '{{.HostConfig.Memory}}' resume-app); swp=$(docker inspect -f '{{.HostConfig.MemorySwap}}' resume-app)
+[[ "$mem" == "$swp" && "$mem" != "0" ]] && ok "memswap_limit == mem_limit ($mem bytes)" || no "mem limits differ ($mem vs $swp)"
+
+# --------------------------------------------------------------- 2. broken build
+hdr "2. build failure fails fast, current does not move"
+before=$(current_target)
+set +e; "$DEPLOY" --source "$SCRATCH/broken" > "$SCRATCH/2-broken.log" 2>&1; rc=$?; set -e
+[[ $rc -ne 0 ]] && ok "deploy.sh exit $rc (non-zero, as required)" || no "broken build reported success"
+[[ "$(current_target)" == "$before" ]] && ok "current unchanged ($before)" || no "current moved after a failed build"
+[[ "$(health)" == "200" ]] && ok "previous release still serving" || no "site down after failed build"
+
+# --------------------------------------------------------------- 3. health gate
+hdr "3. unhealthy release is rolled back by the health gate"
+before=$(current_target)
+set +e; HEALTH_TIMEOUT=10 "$DEPLOY" --source "$SCRATCH/unhealthy" > "$SCRATCH/3-unhealthy.log" 2>&1; rc=$?; set -e
+[[ $rc -ne 0 ]] && ok "deploy.sh exit $rc (non-zero, as required)" || no "unhealthy release reported success"
+[[ "$(current_target)" == "$before" ]] && ok "current restored to $before" || no "current left on $(current_target)"
+[[ "$(health)" == "200" ]] && ok "rolled-back release serving again" || no "no healthy release after rollback"
+grep -q 'HEALTH GATE FAILED' "$SCRATCH/3-unhealthy.log" && ok "log records the health-gate failure" || no "no health-gate failure recorded"
+
+# --------------------------------------------------------------- 4. --rollback
+hdr "4. deploy.sh --rollback restores the previous release in one command"
+"$DEPLOY" --source "$SCRATCH/good" > "$SCRATCH/4-deploy.log" 2>&1 && ok "second good release deployed" || no "second deploy failed"
+v2=$(serving_version)
+prev=$(readlink -f "$ROOT/previous")
+set +e; "$DEPLOY" --rollback > "$SCRATCH/4-rollback.log" 2>&1; rc=$?; set -e
+[[ $rc -eq 0 ]] && ok "rollback exit 0" || no "rollback exit $rc"
+[[ "$(readlink -f "$ROOT/current")" == "$prev" ]] && ok "current flipped back to $prev" || no "current not restored"
+[[ "$(health)" == "200" ]] && ok "rolled-back release healthy" || no "rolled-back release unhealthy"
+v3=$(serving_version)
+[[ -n "$v3" && "$v3" != "$v2" ]] && ok "serving $v3 again (was $v2)" || no "version did not change on rollback"
+
+# --------------------------------------------------------------- 5. restart
+hdr "5. N3: docker compose down && up restores the service"
+rel=$(readlink -f "$ROOT/current")
+docker compose -p resume --project-directory "$rel" -f "$rel/docker-compose.yml" down >/dev/null 2>&1
+[[ "$(health)" != "200" ]] && ok "service down after compose down" || no "still serving after compose down"
+docker compose -p resume --project-directory "$rel" -f "$rel/docker-compose.yml" up -d >/dev/null 2>&1
+sleep 3
+[[ "$(health)" == "200" ]] && ok "service restored after compose up" || no "service did not come back"
+
+# --------------------------------------------------------------- verdict
+hdr "verdict"
+printf 'passed %d, failed %d\n' "$pass" "$fail"
+[[ $fail -eq 0 ]] || exit 1