diff --git a/deploy/Dockerfile b/deploy/Dockerfile new file mode 100644 index 0000000..2321d59 --- /dev/null +++ b/deploy/Dockerfile @@ -0,0 +1,28 @@ +# resume — runtime image (Phase 3 §3.2). +# +# Built by deploy/deploy.sh from a release directory, with the prebuilt static +# Go binary already in place. No Go toolchain, no network and no package install +# at build time: the image is `alpine` + the release payload, nothing else. +# +# The binary must be statically linked (deploy.sh's fallback build sets +# CGO_ENABLED=0). If the app ever needs cgo, change the base image to a glibc +# one (e.g. golang:1.25-bookworm) in git — do not patch it on the host. + +FROM alpine:3.21 + +ARG APP_VERSION=dev +LABEL org.opencontainers.image.title="resume" \ + org.opencontainers.image.version="${APP_VERSION}" \ + org.opencontainers.image.description="Sam Rolfe — portfolio site (Mode A static)" + +WORKDIR /srv + +# bin/server plus whatever the build emitted (dist/ for rendered pages/assets). +COPY . /srv/ + +ENV PORT=8096 \ + APP_VERSION=${APP_VERSION} + +EXPOSE 8096 + +ENTRYPOINT ["/srv/bin/server"] diff --git a/deploy/README.md b/deploy/README.md new file mode 100644 index 0000000..a781934 --- /dev/null +++ b/deploy/README.md @@ -0,0 +1,93 @@ +# Deployment (Phase 3) + +One script, one release directory, one atomic symlink. Deploy is a script, not a +model decision (`AGENTS.md` §4.6), and nobody hand-edits files on the runtime +host: change these files in git and deploy again. + +| File | Purpose | +|---|---| +| `deploy.sh` | build + test → release → flip → `up -d` → health gate → rollback on failure | +| `deploy.sh --rollback` | restore the previous release in one command | +| `docker-compose.yml` | runtime container: `127.0.0.1:8096`, `mem_limit == memswap_limit == 120m`, capped logs | +| `Dockerfile` | runtime image only — alpine + the prebuilt release payload, no toolchain, no network | +| `rehearse.sh` | repeatable proof that deploy, fail-fast, health-gate rollback and `--rollback` all work | + +## Layout on `.13` + +``` +/home/sam/deployment/resume/ +├── releases// immutable: bin/server, dist/, compose, Dockerfile, manifest.txt, .env +├── current -> releases/ live release; swapped with ln + mv -T (atomic rename) +├── previous -> releases/ the release that was live before the last flip +└── deploy.log append-only record of every attempt, including failures +``` + +## What `deploy.sh` expects from the app + +1. `make build` if a `Makefile` with a `build` target exists, otherwise + `go build ./cmd/server`. Either way the result must be one server binary at + `bin/server` or `dist/server`; anything else the binary serves from disk goes + in `dist/` and is copied into the release. +2. `make test` (or `go test ./...`) — runs **before** anything is copied or + flipped, so a broken build never reaches the release directory. +3. `GET /healthz` → `200`. If the body is JSON with a `static` field it must be + `true`. AI readiness is deliberately **not** part of the gate: Mode A must + work with every AI service stopped (acceptance test A1). +4. Bind `$PORT` (default `8096`) on `0.0.0.0` **inside the container**. The + container's own loopback is not reachable through a published port; exposure + is controlled by the host side of `ports:` in `docker-compose.yml`. + +## Deploying + +```bash +cd /home/sam/paseo/projects/resume_ai_system_paperclip_build +./deploy/deploy.sh # build, test, release, flip, health gate +./deploy/deploy.sh --rollback # one command back to the previous release +tail -40 /home/sam/deployment/resume/deploy.log +``` + +Exit code is the answer: `0` means the new release answered `/healthz`. +Non-zero means either (a) build/test failed and **nothing changed**, or (b) the +health gate failed, `current` was flipped back to `previous`, the previous +release was restarted, and the log says which. + +Flags worth knowing: `--source DIR` (build from another checkout — this is how +`rehearse.sh` drives the script), `--root DIR`, `--port N`, +`--health-timeout N`. + +The Go toolchain is auto-detected, including `/nix/store/*-go-1.*/bin/go`, so +the script works on a host where `go` is not on `PATH`. + +## Rehearsing the rollback + +```bash +./deploy/rehearse.sh # 21 assertions, exit 0 when every one holds +``` + +It builds three throwaway services in a scratch directory, runs the real script +against the real release root and the real port, and asserts: + +1. a good release deploys, answers `/healthz`, publishes on `127.0.0.1` only, + and has `memswap_limit == mem_limit`; +2. a build failure exits non-zero, leaves `current` untouched and keeps serving; +3. a release whose `/healthz` is `500` is rolled back by the health gate, exits + non-zero, and the previous release serves again; +4. `deploy.sh --rollback` flips back to the previous release and exits 0; +5. `docker compose down && up` (acceptance test N3) restores the service. + +It removes the release root, container and images it created. `KEEP_REHEARSAL=1` +keeps them for inspection. + +## Container limits — not negotiable + +`mem_limit` and `memswap_limit` are both `120m`, so the container cannot eat +host swap on a host that is already swapping. Measured RSS for a Go static +server is single-digit MiB; `deploy.sh` prints the real figure after every +deploy so drift shows up in the log. + +## Caddy + +`docs/phase-3-deploy.md` §3.3 holds the block for `.35`. It is additive, applied +with `caddy reload` (never a restart), and **Sam applies it** — see +`docs/phase-3-evidence.md` for the open bind-address question that decides +whether that block points at `192.168.20.13:8096` or at `.13`'s Caddy. diff --git a/deploy/deploy.sh b/deploy/deploy.sh new file mode 100755 index 0000000..1ec9efc --- /dev/null +++ b/deploy/deploy.sh @@ -0,0 +1,303 @@ +#!/usr/bin/env bash +# resume — release script (Phase 3). +# +# Deploy is a script, not a model decision (AGENTS.md §4.6). +# No one hand-edits files on the production host: change this script in git. +# +# ./deploy/deploy.sh build, test, release, flip, health-gate +# ./deploy/deploy.sh --rollback restore the previous release, one command +# +# Layout on the runtime host (.13): +# +# /home/sam/deployment/resume/ +# ├── releases// immutable release: bin/, dist/, compose, manifest +# ├── current -> releases/ the live release (atomic symlink flip) +# ├── previous -> releases/ the release that was live before the last flip +# └── deploy.log append-only record of every deploy attempt +# +# A release is only "current" after /healthz answers 200 on 127.0.0.1:$PORT. +# If the health gate fails, `current` is flipped back to `previous`, the previous +# release is restarted, and this script exits non-zero. + +set -Eeuo pipefail + +APP_NAME="resume" +DEFAULT_PORT=8096 +DEFAULT_ROOT="/home/sam/deployment/resume" + +SRC_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +ROOT="${DEPLOY_ROOT:-$DEFAULT_ROOT}" +PORT="${APP_PORT:-$DEFAULT_PORT}" +DO_ROLLBACK=0 +HEALTH_TIMEOUT="${HEALTH_TIMEOUT:-40}" + +usage() { + cat <<'EOF' +Usage: deploy.sh [options] + + --rollback restore the previous release and exit + --source DIR build from DIR instead of the repo this script lives in + --root DIR release root (default /home/sam/deployment/resume) + --port N app port (default 8096) + --health-timeout N seconds to wait for /healthz (default 40) + -h, --help this text + +Environment: + GO path to the go toolchain (auto-detected, incl. /nix/store) + DEPLOY_ROOT same as --root + APP_PORT same as --port +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --rollback) DO_ROLLBACK=1; shift ;; + --source) SRC_DIR="$(cd "$2" && pwd)"; shift 2 ;; + --root) ROOT="$2"; shift 2 ;; + --port) PORT="$2"; shift 2 ;; + --health-timeout) HEALTH_TIMEOUT="$2"; shift 2 ;; + -h|--help) usage; exit 0 ;; + *) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;; + esac +done + +RELEASES="$ROOT/releases" +CURRENT="$ROOT/current" +PREVIOUS="$ROOT/previous" +LOG="$ROOT/deploy.log" + +mkdir -p "$RELEASES" + +log() { printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" | tee -a "$LOG" >&2; } +die() { log "FAIL: $*"; exit 1; } + +# ---------------------------------------------------------------- toolchain +find_go() { + if [[ -n "${GO:-}" && -x "${GO}" ]]; then echo "$GO"; return; fi + if command -v go >/dev/null 2>&1; then command -v go; return; fi + # NixOS: the toolchain lives in the store even when it is not on PATH. + local cand + cand=$(ls -d /nix/store/*-go-1.*/bin/go 2>/dev/null | grep -v '\.drv' | sort -V | tail -1) + if [[ -n "$cand" && -x "$cand" ]]; then echo "$cand"; return; fi + return 1 +} + +compose() { + # $1 = release directory that holds docker-compose.yml + .env + local dir="$1"; shift + docker compose -p "$APP_NAME" --project-directory "$dir" -f "$dir/docker-compose.yml" "$@" +} + +# ---------------------------------------------------------------- health gate +check_health() { + # 200 on /healthz, and if the body is JSON with a "static" field it must be true. + # AI readiness is deliberately NOT part of this gate: Mode A must work with + # every AI service stopped (acceptance test A1). + local url="http://127.0.0.1:${PORT}/healthz" body code + body=$(curl -sS --max-time 3 -w '\n%{http_code}' "$url" 2>/dev/null || true) + code=$(printf '%s' "$body" | tail -1) + body=$(printf '%s' "$body" | sed '$d') + if [[ "$code" != "200" ]]; then + log "health: $url -> HTTP ${code:-none}" + return 1 + fi + if printf '%s' "$body" | head -c 1 | grep -q '{' && command -v jq >/dev/null 2>&1; then + local static_ok + static_ok=$(printf '%s' "$body" | jq -r 'if has("static") then (.static|tostring) else "absent" end' 2>/dev/null || echo absent) + if [[ "$static_ok" == "false" ]]; then + log "health: $url -> 200 but static readiness is false: $body" + return 1 + fi + fi + log "health: $url -> 200" + return 0 +} + +wait_healthy() { + local i + for ((i = 1; i <= HEALTH_TIMEOUT; i++)); do + if check_health; then return 0; fi + sleep 1 + done + return 1 +} + +container_state() { + docker inspect -f 'name={{.Name}} status={{.State.Status}} mem={{.HostConfig.Memory}} memswap={{.HostConfig.MemorySwap}} ports={{json .HostConfig.PortBindings}}' \ + "${APP_NAME}-app" 2>/dev/null || echo "container ${APP_NAME}-app: absent" +} + +# ---------------------------------------------------------------- build + test +discover_server() { + # Contract (deploy/README.md): the app build produces one server binary at + # bin/server or dist/server. `make build` may also emit dist/ for static assets. + local d + for d in "$1/bin/server" "$1/dist/server" "$1/server"; do + if [[ -f "$d" && -x "$d" ]]; then echo "$d"; return 0; fi + done + return 1 +} + +build_and_test() { + local src="$1" out="$2" + local gobin="" + log "build: source $src" + mkdir -p "$out/bin" + cd "$src" + + gobin=$(find_go || true) + if [[ -z "$gobin" ]]; then + log "build: no go toolchain found (set GO=/path/to/go)" + else + export PATH="$(dirname "$gobin"):$PATH" + log "build: go $("$gobin" version)" + fi + + if [[ -f Makefile ]] && grep -qE '^build:' Makefile; then + log "build: make build" + make build + else + [[ -n "$gobin" ]] || die "no Makefile 'build' target and no go toolchain" + log "build: go build ./cmd/server" + CGO_ENABLED=0 "$gobin" build -trimpath -buildvcs=false -o "$out/bin/server" ./cmd/server + fi + + # Tests fail fast, before anything is copied or flipped. + if [[ -f Makefile ]] && grep -qE '^test:' Makefile; then + log "test: make test" + make test + elif [[ -n "$gobin" ]]; then + log "test: go test ./..." + "$gobin" test ./... + fi + + # Collect the binary, wherever the app's build put it. + local bin + bin=$(discover_server "$src" || true) + if [[ -z "$bin" ]]; then + bin=$(discover_server "$out" || true) + fi + [[ -n "$bin" ]] || die "no server binary found (looked for bin/server, dist/server, server)" + if [[ "$(readlink -f "$bin")" != "$(readlink -f "$out/bin/server")" ]]; then + cp -f "$bin" "$out/bin/server" + fi + chmod +x "$out/bin/server" + log "build: binary $(stat -c %s "$out/bin/server") bytes, sha256 $(sha256sum "$out/bin/server" | cut -c1-16)" + + # Static assets, if the build emits them. + if [[ -d "$src/dist" ]]; then + cp -a "$src/dist/." "$out/dist/" 2>/dev/null || { mkdir -p "$out/dist"; cp -a "$src/dist/." "$out/dist/"; } + log "build: copied dist/ ($(find "$out/dist" -type f | wc -l) files)" + fi +} + +# ---------------------------------------------------------------- deploy +do_deploy() { + local src="$1" + local ts sha branch release prev_target + ts=$(date -u +%Y%m%dT%H%M%SZ) + sha=$(git -C "$src" rev-parse --short HEAD 2>/dev/null || echo unknown) + branch=$(git -C "$src" rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown) + release="$RELEASES/$ts" + + log "deploy: $APP_NAME $ts (branch $branch, $sha) -> $release" + mkdir -p "$release" + + # 1. Build + test. Nothing is touched on failure. + build_and_test "$src" "$release" + + # 2. Release payload: binary, compose file, runtime image, manifest. + cp -f "$src/deploy/docker-compose.yml" "$release/docker-compose.yml" + cp -f "$src/deploy/Dockerfile" "$release/Dockerfile" + { + echo "app=$APP_NAME" + echo "release=$ts" + echo "git_sha=$(git -C "$src" rev-parse HEAD 2>/dev/null || echo unknown)" + echo "git_branch=$branch" + echo "built_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "built_on=$(hostname)" + echo "binary_sha256=$(sha256sum "$release/bin/server" | cut -d' ' -f1)" + } > "$release/manifest.txt" + printf 'APP_VERSION=%s\nAPP_PORT=%s\n' "$ts" "$PORT" > "$release/.env" + + # 3. Runtime image, tagged with this release. + log "deploy: docker build resume/app:$ts" + docker build --quiet -t "resume/app:$ts" \ + --build-arg "APP_VERSION=$ts" -f "$release/Dockerfile" "$release" >>"$LOG" 2>&1 \ + || die "image build failed" + + # 4. Remember what is live now, then flip atomically. + prev_target="" + if [[ -L "$CURRENT" ]]; then prev_target=$(readlink -f "$CURRENT"); fi + ln -sfn "$release" "$CURRENT.tmp" && mv -T "$CURRENT.tmp" "$CURRENT" || die "symlink flip failed" + if [[ -n "$prev_target" && "$prev_target" != "$(readlink -f "$release")" ]]; then + ln -sfn "$prev_target" "$PREVIOUS.tmp" && mv -T "$PREVIOUS.tmp" "$PREVIOUS" + log "deploy: previous -> $prev_target" + fi + log "deploy: current -> $(readlink -f "$CURRENT")" + + # 5. Bring up, then gate on health. Anything less than 200 means roll back. + compose "$release" up -d --remove-orphans >>"$LOG" 2>&1 || log "deploy: compose up returned non-zero" + if ! wait_healthy; then + log "deploy: HEALTH GATE FAILED for $ts" + rollback_internal "$ts" + exit 1 + fi + + log "deploy: OK $ts" + container_state | tee -a "$LOG" >&2 + docker stats --no-stream --format 'rss={{.MemUsage}} pct={{.MemPerc}}' "${APP_NAME}-app" 2>/dev/null | tee -a "$LOG" >&2 || true +} + +rollback_internal() { + # Used by a failed deploy: restore the previous release, restart it, verify. + local failed="$1" target + if [[ ! -L "$PREVIOUS" ]]; then + log "rollback: no previous release to restore; stopping ${APP_NAME}-app" + compose "$CURRENT" down >>"$LOG" 2>&1 || true + return 1 + fi + target=$(readlink -f "$PREVIOUS") + log "rollback: $failed failed -> restoring $target" + ln -sfn "$target" "$CURRENT.tmp" && mv -T "$CURRENT.tmp" "$CURRENT" + compose "$target" up -d --remove-orphans >>"$LOG" 2>&1 || true + if wait_healthy; then + log "rollback: previous release healthy ($(readlink -f "$CURRENT"))" + else + log "rollback: previous release ALSO unhealthy — manual attention required" + return 1 + fi +} + +do_rollback() { + local target + if [[ ! -L "$PREVIOUS" ]]; then die "no previous release at $PREVIOUS — nothing to roll back to"; fi + target=$(readlink -f "$PREVIOUS") + [[ -d "$target" ]] || die "previous release target missing: $target" + log "rollback: current=$( [[ -L "$CURRENT" ]] && readlink -f "$CURRENT" || echo none) -> $target" + + # Keep the release we are leaving, so the rollback is itself reversible. + local leaving="" + if [[ -L "$CURRENT" ]]; then leaving=$(readlink -f "$CURRENT"); fi + ln -sfn "$target" "$CURRENT.tmp" && mv -T "$CURRENT.tmp" "$CURRENT" || die "symlink flip failed" + if [[ -n "$leaving" && "$leaving" != "$target" ]]; then + ln -sfn "$leaving" "$PREVIOUS.tmp" && mv -T "$PREVIOUS.tmp" "$PREVIOUS" + fi + + compose "$target" up -d --remove-orphans >>"$LOG" 2>&1 || log "rollback: compose up returned non-zero" + if wait_healthy; then + log "rollback: OK, current -> $target" + container_state | tee -a "$LOG" >&2 + exit 0 + fi + log "rollback: FAILED, $target is not healthy" + exit 1 +} + +mkdir -p "$ROOT" +log "=== ${APP_NAME} deploy.sh start (pid $$) ===" +if (( DO_ROLLBACK == 1 )); then + do_rollback +else + do_deploy "$SRC_DIR" +fi diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml new file mode 100644 index 0000000..d1c7978 --- /dev/null +++ b/deploy/docker-compose.yml @@ -0,0 +1,41 @@ +# resume — runtime container (Phase 3 §3.2). +# +# Copied into each release directory by deploy/deploy.sh; the release's .env +# supplies APP_VERSION (the image tag). Never edited on the host — change it +# here, in git, and deploy again. +# +# Limits that are not negotiable (AGENTS.md §4.3): +# mem_limit == memswap_limit == 120m -> the container cannot touch host swap +# ports 127.0.0.1:$PORT:8096 -> Caddy on .35 is the only public listener +# +# .13 has every default Docker address pool subnetted out (many projects), so a +# single-container service uses the host bridge and publishes its port rather +# than creating a project network. Same pattern as compass-teams-nhs. + +name: resume + +services: + app: + image: resume/app:${APP_VERSION} + container_name: resume-app + restart: unless-stopped + network_mode: bridge + ports: + - "127.0.0.1:${APP_PORT:-8096}:8096" + environment: + PORT: "8096" + APP_VERSION: "${APP_VERSION}" + # Mode B values arrive in Phase 5 from ~/.config/environment.d/ — never here. + mem_limit: 120m + memswap_limit: 120m + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" + healthcheck: + test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8096/healthz"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 5s diff --git a/deploy/rehearse.sh b/deploy/rehearse.sh new file mode 100755 index 0000000..812aebe --- /dev/null +++ b/deploy/rehearse.sh @@ -0,0 +1,184 @@ +#!/usr/bin/env bash +# resume — rollback rehearsal (Phase 3 DoD: "Rollback rehearsed once and documented"). +# +# Builds three throwaway Go services in a scratch directory and runs the real +# deploy mechanism against the real release root and the real port: +# +# 1. good -> deploys, health-gated, serving +# 2. broken -> does not compile: deploy must fail fast, current must not move +# 3. unhealthy -> builds and runs, but /healthz is 500: the health gate must +# roll back to the previous release and exit non-zero +# 4. good again -> then `deploy.sh --rollback` must restore the earlier release +# +# It leaves nothing running and removes the release root it created, unless +# KEEP_REHEARSAL=1. Nothing outside $DEPLOY_ROOT, the scratch dir and this +# service's own container/image is touched. +# +# ./deploy/rehearse.sh +# +# Exit 0 means every assertion held; the transcript is the evidence. + +set -Eeuo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +DEPLOY="$HERE/deploy.sh" +ROOT="${DEPLOY_ROOT:-/home/sam/deployment/resume}" +PORT="${APP_PORT:-8096}" +SCRATCH="${REHEARSAL_DIR:-$(mktemp -d -t resume-rehearsal-XXXXXX)}" +KEEP="${KEEP_REHEARSAL:-0}" + +pass=0; fail=0 +ok() { printf ' PASS %s\n' "$*"; pass=$((pass + 1)); } +no() { printf ' FAIL %s\n' "$*"; fail=$((fail + 1)); } +hdr() { printf '\n== %s\n' "$*"; } + +cleanup() { + if docker ps -a --format '{{.Names}}' | grep -qx 'resume-app'; then + docker rm -f resume-app >/dev/null 2>&1 || true + fi + docker images --format '{{.Repository}}:{{.Tag}}' | grep -q '^resume/app:' \ + && docker images --format '{{.Repository}}:{{.Tag}}' | grep '^resume/app:' \ + | xargs -r docker rmi -f >/dev/null 2>&1 || true + if [[ "$KEEP" == "1" ]]; then + printf '\nkept: release root %s, scratch %s, images resume/app:*\n' "$ROOT" "$SCRATCH" + else + rm -rf "$ROOT" + [[ "$SCRATCH" == /tmp/resume-rehearsal-* || -n "${REHEARSAL_DIR:-}" ]] && rm -rf "$SCRATCH" + printf '\ncleaned: release root and rehearsal sources removed\n' + fi +} +trap cleanup EXIT + +# --------------------------------------------------------------- fixtures +make_app() { # $1 dir, $2 mode: good|broken|unhealthy + local dir="$1" mode="$2" + mkdir -p "$dir/cmd/server" "$dir/deploy" + cp -f "$HERE/docker-compose.yml" "$HERE/Dockerfile" "$dir/deploy/" + printf 'module rehearsal\n\ngo 1.25\n' > "$dir/go.mod" + cat > "$dir/cmd/server/main.go" <<'GO' +package main + +import ( + "fmt" + "log" + "net/http" + "os" + "time" +) + +func main() { + port := os.Getenv("PORT") + if port == "" { + port = "8096" + } + bind := os.Getenv("BIND") + if bind == "" { + bind = "0.0.0.0" + } + mux := http.NewServeMux() + mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { + BROKEN + w.Header().Set("Content-Type", "application/json") + fmt.Fprintf(w, `{"status":"ok","static":true,"ai":"disabled","version":%q,"time":%q}`, + os.Getenv("APP_VERSION"), time.Now().UTC().Format(time.RFC3339)) + }) + mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + fmt.Fprintf(w, "rehearsal

version %s

", os.Getenv("APP_VERSION")) + }) + srv := &http.Server{Addr: bind + ":" + port, Handler: mux, ReadHeaderTimeout: 5 * time.Second} + log.Printf("listening on %s:%s", bind, port) + if err := srv.ListenAndServe(); err != nil { + log.Fatal(err) + } +} +GO + local patch='' + case "$mode" in + good) patch='' ;; + unhealthy) patch='w.WriteHeader(http.StatusInternalServerError)' ;; + broken) patch='' ;; + esac + python3 - "$dir/cmd/server/main.go" "$patch" <<'PY' +import sys +p, patch = sys.argv[1], sys.argv[2] +s = open(p).read().replace("BROKEN", patch, 1) +open(p, "w").write(s) +PY + if [[ "$mode" == "broken" ]]; then + sed -i 's/^func main() {/func main( {/' "$dir/cmd/server/main.go" + fi +} + +health() { curl -s -m 3 -o /dev/null -w '%{http_code}' "http://127.0.0.1:$PORT/healthz" || true; } +serving_version() { curl -s -m 3 "http://127.0.0.1:$PORT/healthz" | sed -n 's/.*"version":"\([^"]*\)".*/\1/p'; } +current_target() { [[ -L "$ROOT/current" ]] && readlink -f "$ROOT/current" || echo none; } + +# --------------------------------------------------------------- 0. preconditions +hdr "0. preconditions" +if ss -ltn 2>/dev/null | awk '{print $4}' | grep -q ":$PORT\$"; then + no "port $PORT already in use — nothing started" + exit 1 +fi +ok "port $PORT free" +rm -rf "$ROOT" +make_app "$SCRATCH/good" good +make_app "$SCRATCH/broken" broken +make_app "$SCRATCH/unhealthy" unhealthy +ok "three throwaway sources built in $SCRATCH" + +# --------------------------------------------------------------- 1. good deploy +hdr "1. good release deploys and health-gates" +if "$DEPLOY" --source "$SCRATCH/good" > "$SCRATCH/1-deploy.log" 2>&1; then + ok "deploy.sh exit 0" +else + no "deploy.sh exit non-zero"; tail -20 "$SCRATCH/1-deploy.log"; exit 1 +fi +[[ "$(health)" == "200" ]] && ok "/healthz 200 on 127.0.0.1:$PORT" || no "/healthz not 200" +v1=$(serving_version); [[ -n "$v1" ]] && ok "serving release $v1" || no "no version reported" +ss -ltn | grep -q "127.0.0.1:$PORT" && ok "container publishes on 127.0.0.1 only" || no "unexpected bind" +mem=$(docker inspect -f '{{.HostConfig.Memory}}' resume-app); swp=$(docker inspect -f '{{.HostConfig.MemorySwap}}' resume-app) +[[ "$mem" == "$swp" && "$mem" != "0" ]] && ok "memswap_limit == mem_limit ($mem bytes)" || no "mem limits differ ($mem vs $swp)" + +# --------------------------------------------------------------- 2. broken build +hdr "2. build failure fails fast, current does not move" +before=$(current_target) +set +e; "$DEPLOY" --source "$SCRATCH/broken" > "$SCRATCH/2-broken.log" 2>&1; rc=$?; set -e +[[ $rc -ne 0 ]] && ok "deploy.sh exit $rc (non-zero, as required)" || no "broken build reported success" +[[ "$(current_target)" == "$before" ]] && ok "current unchanged ($before)" || no "current moved after a failed build" +[[ "$(health)" == "200" ]] && ok "previous release still serving" || no "site down after failed build" + +# --------------------------------------------------------------- 3. health gate +hdr "3. unhealthy release is rolled back by the health gate" +before=$(current_target) +set +e; HEALTH_TIMEOUT=10 "$DEPLOY" --source "$SCRATCH/unhealthy" > "$SCRATCH/3-unhealthy.log" 2>&1; rc=$?; set -e +[[ $rc -ne 0 ]] && ok "deploy.sh exit $rc (non-zero, as required)" || no "unhealthy release reported success" +[[ "$(current_target)" == "$before" ]] && ok "current restored to $before" || no "current left on $(current_target)" +[[ "$(health)" == "200" ]] && ok "rolled-back release serving again" || no "no healthy release after rollback" +grep -q 'HEALTH GATE FAILED' "$SCRATCH/3-unhealthy.log" && ok "log records the health-gate failure" || no "no health-gate failure recorded" + +# --------------------------------------------------------------- 4. --rollback +hdr "4. deploy.sh --rollback restores the previous release in one command" +"$DEPLOY" --source "$SCRATCH/good" > "$SCRATCH/4-deploy.log" 2>&1 && ok "second good release deployed" || no "second deploy failed" +v2=$(serving_version) +prev=$(readlink -f "$ROOT/previous") +set +e; "$DEPLOY" --rollback > "$SCRATCH/4-rollback.log" 2>&1; rc=$?; set -e +[[ $rc -eq 0 ]] && ok "rollback exit 0" || no "rollback exit $rc" +[[ "$(readlink -f "$ROOT/current")" == "$prev" ]] && ok "current flipped back to $prev" || no "current not restored" +[[ "$(health)" == "200" ]] && ok "rolled-back release healthy" || no "rolled-back release unhealthy" +v3=$(serving_version) +[[ -n "$v3" && "$v3" != "$v2" ]] && ok "serving $v3 again (was $v2)" || no "version did not change on rollback" + +# --------------------------------------------------------------- 5. restart +hdr "5. N3: docker compose down && up restores the service" +rel=$(readlink -f "$ROOT/current") +docker compose -p resume --project-directory "$rel" -f "$rel/docker-compose.yml" down >/dev/null 2>&1 +[[ "$(health)" != "200" ]] && ok "service down after compose down" || no "still serving after compose down" +docker compose -p resume --project-directory "$rel" -f "$rel/docker-compose.yml" up -d >/dev/null 2>&1 +sleep 3 +[[ "$(health)" == "200" ]] && ok "service restored after compose up" || no "service did not come back" + +# --------------------------------------------------------------- verdict +hdr "verdict" +printf 'passed %d, failed %d\n' "$pass" "$fail" +[[ $fail -eq 0 ]] || exit 1 diff --git a/docs/phase-3-evidence.md b/docs/phase-3-evidence.md new file mode 100644 index 0000000..8c86b09 --- /dev/null +++ b/docs/phase-3-evidence.md @@ -0,0 +1,231 @@ +# Phase 3 evidence — public deployment + +**Status: mechanism complete and rehearsed; public URL deferred.** +The deploy machinery exists, is rehearsed against the real release root and the +real port on `.13`, and leaves nothing running. The site itself is not deployed +because Phase 2 has not produced a binary yet, and one decision is needed from +Sam before the container can be reachable by Caddy on `.35`. + +Recorded 2026-10-26 on `.13` (`nixos-desktop`), run `630e2ed9`. + +--- + +## Definition of Done + +| # | Item | Verdict | Evidence | +|---|---|---|---| +| 1 | `deploy.sh` and `deploy.sh --rollback` work; a deliberate bad build proves rollback | **met** | §1, `deploy/rehearse.sh` 21/21 | +| 2 | `https://resume.lab.audasmedia.com.au` serves the site | **deferred** | Phase 2 has not produced a binary; Caddy block not yet applied (§4, §5) | +| 3 | Container bound to `127.0.0.1` only, `ss -ltn` | **met for the container**; conflicts with reachability from `.35` — see §5 | §2 | +| 4 | `memswap_limit == mem_limit`, `docker inspect` | **met** | §2 | +| 5 | Security headers present, CSP does not break the page | **deferred** | headers belong to the `.35` Caddy block (§4) | +| 6 | A1 passes over the public URL with the AI layer stopped | **deferred** | needs a live URL | +| 7 | Sam applied the Caddy block and confirmed other lab hostnames | **deferred — Sam's action** | block and verification steps in §4 | +| 8 | Rollback rehearsed once and documented | **met** | §1, `deploy/rehearse.sh` | + +Nothing in the deferred column is asserted. Each depends on the Phase 2 binary +(items 2, 5, 6) or on Sam (items 2, 5, 7). + +--- + +## 1. What was rehearsed, and the result + +`deploy/rehearse.sh` builds throwaway services and drives the real script. Full +transcript: + +``` +== 0. preconditions + PASS port 8096 free + PASS three throwaway sources built in /tmp/.../resume-rehearsal-snpEjy + +== 1. good release deploys and health-gates + PASS deploy.sh exit 0 + PASS /healthz 200 on 127.0.0.1:8096 + PASS serving release 20261026T062957Z + PASS container publishes on 127.0.0.1 only + PASS memswap_limit == mem_limit (125829120 bytes) + +== 2. build failure fails fast, current does not move + PASS deploy.sh exit 1 (non-zero, as required) + PASS current unchanged (/home/sam/deployment/resume/releases/20261026T062957Z) + PASS previous release still serving + +== 3. unhealthy release is rolled back by the health gate + PASS deploy.sh exit 1 (non-zero, as required) + PASS current restored to /home/sam/deployment/resume/releases/20261026T062957Z + PASS rolled-back release serving again + PASS log records the health-gate failure + +== 4. deploy.sh --rollback restores the previous release in one command + PASS second good release deployed + PASS rollback exit 0 + PASS current flipped back to /home/sam/deployment/resume/releases/20261026T062957Z + PASS rolled-back release healthy + PASS serving 20261026T062957Z again (was 20261026T063021Z) + +== 5. N3: docker compose down && up restores the service + PASS service down after compose down + PASS service restored after compose up + +== verdict +passed 21, failed 0 +``` + +The health-gate rollback in the deploy log, verbatim: + +``` +06:28:46Z deploy: previous -> /home/sam/deployment/resume/releases/20261026T062828Z +06:28:46Z deploy: current -> /home/sam/deployment/resume/releases/20261026T062842Z +06:29:02Z deploy: HEALTH GATE FAILED for 20261026T062842Z +06:29:02Z rollback: 20261026T062842Z failed -> restoring .../releases/20261026T062828Z +06:29:04Z health: http://127.0.0.1:8096/healthz -> 200 +06:29:04Z rollback: previous release healthy (.../releases/20261026T062828Z) +``` + +The deliberate bad build, verbatim: + +``` +$ ./deploy/deploy.sh --source .../app-broken +cmd/server/main.go:11:12: syntax error: unexpected {, expected ) +exit=1 +current before: .../releases/20261026T062828Z +current after: .../releases/20261026T062828Z # unchanged +health after broken build: 200 # old release still serving +``` + +Re-run it any time with `./deploy/rehearse.sh` (exit 0 = every assertion held). + +--- + +## 2. Container limits and bind scope + +``` +$ ss -ltn | grep 8096 +LISTEN 0 4096 127.0.0.1:8096 0.0.0.0:* + +$ docker inspect -f '...' resume-app +name=/resume-app +mem_limit=125829120 bytes # 120 MiB +memswap_limit=125829120 bytes # 120 MiB — equal, so no host swap +port_bindings={"8096/tcp":[{"HostIp":"127.0.0.1","HostPort":"8096"}]} +restart=unless-stopped +log_config={"Type":"json-file","Config":{"max-file":"3","max-size":"10m"}} +``` + +Measured memory (idle Go static server, placeholder build): + +``` +rss=1.5MiB / 120MiB pct=1.25% +rss=6.9MiB / 120MiB pct=5.78% # just after start, before settling +``` + +Host pressure at the time of the rehearsal — this is why the cap is 120m: + +``` +$ free -m + total used free available +Mem: 15918 14255 227 1663 +``` + +No residue was left behind: after the rehearsal there is no `resume-app` +container, no `resume/app:*` image, nothing listening on 8096, and +`/home/sam/deployment/resume/` does not exist. `deploy.sh` creates it on the +first real deploy. (The pre-existing `ai-resume-*` containers from +`/home/sam/deployment/ai-resume/` were not touched.) + +--- + +## 3. Facts established on the host + +| Fact | Value | +|---|---| +| Port 8096 | free (`ss -ltn`; 8090, 8091, 8092, 8095 are taken) | +| Release root | `/home/sam/deployment/resume/` — created by `deploy.sh` | +| Go toolchain | not on `PATH`; `/nix/store/…-go-1.26.5/bin/go`, auto-detected by the script | +| Container runtime | Docker 29.6.2, Compose 5.3.1 | +| `.13` runs its own Caddy | `caddy run --config /etc/static/caddy/caddy_config`, listening `*:8000`, 8 legacy site blocks, all `reverse_proxy localhost:` | +| Services added since then | bind `0.0.0.0` and are proxied by `.35` directly on their own LAN port (hub 8788, compass 8792, email-agent 8790, webrestrict 8791) | + +--- + +## 4. Caddy block for Sam to apply (`.35`, additive, `caddy reload` only) + +```caddy +resume.lab.audasmedia.com.au { + encode zstd gzip + reverse_proxy 192.168.20.13:8096 + header { + Strict-Transport-Security "max-age=31536000; includeSubDomains" + X-Content-Type-Options "nosniff" + Referrer-Policy "strict-origin-when-cross-origin" + X-Frame-Options "SAMEORIGIN" + Content-Security-Policy "default-src 'self'; img-src 'self' data:; script-src 'self'; style-src 'self' 'unsafe-inline'; connect-src 'self'; frame-ancestors 'none'; base-uri 'none'" + } + @static path *.css *.js *.svg *.png *.jpg *.webp *.woff2 + header @static Cache-Control "public, max-age=31536000, immutable" +} +``` + +Apply, in this order: + +```bash +# 1. add the block above to the Caddyfile on .35, then validate: +caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile +# 2. reload — not restart, so other sites are not interrupted: +caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile +# 3. confirm the other hostnames first, from anywhere on the LAN: +for h in gitea.lab.audasmedia.com.au immich.lab.audasmedia.com.au \ + nextcloud.lab.audasmedia.com.au penpot.lab.audasmedia.com.au \ + wiki.lab.audasmedia.com.au chat.lab.audasmedia.com.au; do + printf '%-45s %s\n' "$h" "$(curl -s -o /dev/null -w '%{http_code}' "https://$h/")" +done +# 4. then the new one (404/502 is expected until Phase 2's binary is deployed): +curl -sSI https://resume.lab.audasmedia.com.au/ | sed -n '1,12p' +``` + +Header check after that (`item 5` of the DoD): the response above must include +`Strict-Transport-Security`, `X-Content-Type-Options`, `Referrer-Policy`, +`X-Frame-Options` and `Content-Security-Policy`. If the page renders but the CSP +breaks it, the CSP is wrong — do not weaken it to `unsafe-inline` for scripts. + +--- + +## 5. Open question for Sam — bind address vs Caddy reachability + +The phase document says two things that cannot both be true today: + +- §3.2 and DoD item 3: the container publishes on **`127.0.0.1:8096` only**. +- §3.3: Caddy **on `.35`** proxies to **`192.168.20.13:8096`**. + +A loopback-only listener on `.13` is not reachable from `.35`. Measured: + +``` +$ curl -sS -m3 http://192.168.20.13:8096/healthz +curl: (7) Failed to connect to 192.168.20.13:8096 +``` + +`.13`'s own Caddy (`*:8000`, `/etc/static/caddy/caddy_config`) only serves eight +legacy hostnames; every service added since (hub, compass, email-agent, +webrestrict) publishes on `0.0.0.0` and is proxied by `.35` directly to its LAN +port. That is the working pattern, and it is what §3.3's block assumes. + +| Option | What changes | Consequence | +|---|---|---| +| **A (recommended, matches every current service)** | publish `127.0.0.1:8096` **and** `192.168.20.13:8096` | `deploy.sh` keeps its loopback health gate; `.35` reaches the app; §3.3 applied verbatim; no NixOS change. Exposure is the `.13` LAN, same as hub/compass today. DoD item 3 / N1 can then only claim "not on `0.0.0.0`" — needs your written change | +| **B (strict loopback)** | add `http://resume.lab.audasmedia.com.au:8000 { reverse_proxy localhost:8096 }` to `.13`'s Caddyfile, and `.35`'s block becomes `reverse_proxy 192.168.20.13:8000` | Satisfies N1 literally. But `/etc/static/caddy/caddy_config` is NixOS-generated — it needs a NixOS config change and `nixos-rebuild`, which this phase forbids and only you can do | + +The committed `docker-compose.yml` currently follows the phase document (option +**A**'s loopback half only: `127.0.0.1:8096`). Choosing A or B is a one-line +change in `deploy/docker-compose.yml` plus, for B, the NixOS change on `.13`. + +--- + +## 6. Deferred, with owners + +| Item | Owner | Unblocked by | +|---|---|---| +| Deploy the real site (DoD 2) | this issue, woken by [AUD-132](/AUD/issues/AUD-132) | Phase 2 producing `bin/server` + `dist/` from `make build` | +| Header + CSP proof (DoD 5) | this issue, after the Caddy block is applied | Sam applies §4 | +| A1 over the public URL (DoD 6) | this issue, after 2 and 5 | — | +| Caddy block applied and other hostnames confirmed (DoD 7) | **Sam** | §4 | +| Bind-address decision (§5) | **Sam** | A or B above |