ai-resume/docs/phase-3-evidence.md

Phase 3 evidence — public deployment

Status: mechanism complete and rehearsed; public URL deferred. The deploy machinery exists, is rehearsed against the real release root and the real port on .13, and leaves nothing running. The site itself is not deployed because Phase 2 has not produced a binary yet, and one decision is needed from Sam before the container can be reachable by Caddy on .35.

Recorded 2026-10-26 on .13 (nixos-desktop), run 630e2ed9.

---

Definition of Done

#	Item	Verdict	Evidence
1	deploy.sh and deploy.sh --rollback work; a deliberate bad build proves rollback	met	§1, deploy/rehearse.sh 21/21
2	https://resume.lab.audasmedia.com.au serves the site	deferred	Phase 2 has not produced a binary; Caddy block not yet applied (§4, §5)
3	Container bound to 127.0.0.1 only, ss -ltn	met for the container; conflicts with reachability from .35 — see §5	§2
4	memswap_limit == mem_limit, docker inspect	met	§2
5	Security headers present, CSP does not break the page	deferred	headers belong to the .35 Caddy block (§4)
6	A1 passes over the public URL with the AI layer stopped	deferred	needs a live URL
7	Sam applied the Caddy block and confirmed other lab hostnames	deferred — Sam's action	block and verification steps in §4
8	Rollback rehearsed once and documented	met	§1, deploy/rehearse.sh

Nothing in the deferred column is asserted. Each depends on the Phase 2 binary (items 2, 5, 6) or on Sam (items 2, 5, 7).

---

1. What was rehearsed, and the result

deploy/rehearse.sh builds throwaway services and drives the real script. Full transcript:

== 0. preconditions
  PASS  port 8096 free
  PASS  three throwaway sources built in /tmp/.../resume-rehearsal-snpEjy

== 1. good release deploys and health-gates
  PASS  deploy.sh exit 0
  PASS  /healthz 200 on 127.0.0.1:8096
  PASS  serving release 20261026T062957Z
  PASS  container publishes on 127.0.0.1 only
  PASS  memswap_limit == mem_limit (125829120 bytes)

== 2. build failure fails fast, current does not move
  PASS  deploy.sh exit 1 (non-zero, as required)
  PASS  current unchanged (/home/sam/deployment/resume/releases/20261026T062957Z)
  PASS  previous release still serving

== 3. unhealthy release is rolled back by the health gate
  PASS  deploy.sh exit 1 (non-zero, as required)
  PASS  current restored to /home/sam/deployment/resume/releases/20261026T062957Z
  PASS  rolled-back release serving again
  PASS  log records the health-gate failure

== 4. deploy.sh --rollback restores the previous release in one command
  PASS  second good release deployed
  PASS  rollback exit 0
  PASS  current flipped back to /home/sam/deployment/resume/releases/20261026T062957Z
  PASS  rolled-back release healthy
  PASS  serving 20261026T062957Z again (was 20261026T063021Z)

== 5. N3: docker compose down && up restores the service
  PASS  service down after compose down
  PASS  service restored after compose up

== verdict
passed 21, failed 0

The health-gate rollback in the deploy log, verbatim:

06:28:46Z deploy: previous -> /home/sam/deployment/resume/releases/20261026T062828Z
06:28:46Z deploy: current  -> /home/sam/deployment/resume/releases/20261026T062842Z
06:29:02Z deploy: HEALTH GATE FAILED for 20261026T062842Z
06:29:02Z rollback: 20261026T062842Z failed -> restoring .../releases/20261026T062828Z
06:29:04Z health: http://127.0.0.1:8096/healthz -> 200
06:29:04Z rollback: previous release healthy (.../releases/20261026T062828Z)

The deliberate bad build, verbatim:

$ ./deploy/deploy.sh --source .../app-broken
cmd/server/main.go:11:12: syntax error: unexpected {, expected )
exit=1
current before: .../releases/20261026T062828Z
current after:  .../releases/20261026T062828Z      # unchanged
health after broken build: 200                     # old release still serving

Re-run it any time with ./deploy/rehearse.sh (exit 0 = every assertion held).

---

2. Container limits and bind scope

$ ss -ltn | grep 8096
LISTEN 0  4096  127.0.0.1:8096  0.0.0.0:*

$ docker inspect -f '...' resume-app
name=/resume-app
mem_limit=125829120 bytes          # 120 MiB
memswap_limit=125829120 bytes      # 120 MiB — equal, so no host swap
port_bindings={"8096/tcp":[{"HostIp":"127.0.0.1","HostPort":"8096"}]}
restart=unless-stopped
log_config={"Type":"json-file","Config":{"max-file":"3","max-size":"10m"}}

Measured memory (idle Go static server, placeholder build):

rss=1.5MiB / 120MiB pct=1.25%
rss=6.9MiB / 120MiB pct=5.78%    # just after start, before settling

Host pressure at the time of the rehearsal — this is why the cap is 120m:

$ free -m
               total   used   free   available
Mem:           15918  14255    227        1663

No residue was left behind: after the rehearsal there is no resume-app container, no resume/app:* image, nothing listening on 8096, and /home/sam/deployment/resume/ does not exist. deploy.sh creates it on the first real deploy. (The pre-existing ai-resume-* containers from /home/sam/deployment/ai-resume/ were not touched.)

---

3. Facts established on the host

Fact	Value
Port 8096	free (ss -ltn; 8090, 8091, 8092, 8095 are taken)
Release root	/home/sam/deployment/resume/ — created by deploy.sh
Go toolchain	not on PATH; /nix/store/…-go-1.26.5/bin/go, auto-detected by the script
Container runtime	Docker 29.6.2, Compose 5.3.1
.13 runs its own Caddy	caddy run --config /etc/static/caddy/caddy_config, listening *:8000, 8 legacy site blocks, all reverse_proxy localhost:<port>
Services added since then	bind 0.0.0.0 and are proxied by .35 directly on their own LAN port (hub 8788, compass 8792, email-agent 8790, webrestrict 8791)

---

4. Caddy block for Sam to apply (.35, additive, caddy reload only)

resume.lab.audasmedia.com.au {
	encode zstd gzip
	reverse_proxy 192.168.20.13:8096
	header {
		Strict-Transport-Security "max-age=31536000; includeSubDomains"
		X-Content-Type-Options "nosniff"
		Referrer-Policy "strict-origin-when-cross-origin"
		X-Frame-Options "SAMEORIGIN"
		Content-Security-Policy "default-src 'self'; img-src 'self' data:; script-src 'self'; style-src 'self' 'unsafe-inline'; connect-src 'self'; frame-ancestors 'none'; base-uri 'none'"
	}
	@static path *.css *.js *.svg *.png *.jpg *.webp *.woff2
	header @static Cache-Control "public, max-age=31536000, immutable"
}

Apply, in this order:

# 1. add the block above to the Caddyfile on .35, then validate:
caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
# 2. reload — not restart, so other sites are not interrupted:
caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile
# 3. confirm the other hostnames first, from anywhere on the LAN:
for h in gitea.lab.audasmedia.com.au immich.lab.audasmedia.com.au \
         nextcloud.lab.audasmedia.com.au penpot.lab.audasmedia.com.au \
         wiki.lab.audasmedia.com.au chat.lab.audasmedia.com.au; do
  printf '%-45s %s\n' "$h" "$(curl -s -o /dev/null -w '%{http_code}' "https://$h/")"
done
# 4. then the new one (404/502 is expected until Phase 2's binary is deployed):
curl -sSI https://resume.lab.audasmedia.com.au/ | sed -n '1,12p'

Header check after that (item 5 of the DoD): the response above must include Strict-Transport-Security, X-Content-Type-Options, Referrer-Policy, X-Frame-Options and Content-Security-Policy. If the page renders but the CSP breaks it, the CSP is wrong — do not weaken it to unsafe-inline for scripts.

---

5. Open question for Sam — bind address vs Caddy reachability

The phase document says two things that cannot both be true today:

• §3.2 and DoD item 3: the container publishes on 127.0.0.1:8096 only.
• §3.3: Caddy on .35 proxies to 192.168.20.13:8096.

A loopback-only listener on .13 is not reachable from .35. Measured:

$ curl -sS -m3 http://192.168.20.13:8096/healthz
curl: (7) Failed to connect to 192.168.20.13:8096

.13's own Caddy (*:8000, /etc/static/caddy/caddy_config) only serves eight legacy hostnames; every service added since (hub, compass, email-agent, webrestrict) publishes on 0.0.0.0 and is proxied by .35 directly to its LAN port. That is the working pattern, and it is what §3.3's block assumes.

Option	What changes	Consequence
A (recommended, matches every current service)	publish 127.0.0.1:8096 and 192.168.20.13:8096	deploy.sh keeps its loopback health gate; .35 reaches the app; §3.3 applied verbatim; no NixOS change. Exposure is the .13 LAN, same as hub/compass today. DoD item 3 / N1 can then only claim "not on 0.0.0.0" — needs your written change
B (strict loopback)	add http://resume.lab.audasmedia.com.au:8000 { reverse_proxy localhost:8096 } to .13's Caddyfile, and .35's block becomes reverse_proxy 192.168.20.13:8000	Satisfies N1 literally. But /etc/static/caddy/caddy_config is NixOS-generated — it needs a NixOS config change and nixos-rebuild, which this phase forbids and only you can do

The committed docker-compose.yml currently follows the phase document (option A's loopback half only: 127.0.0.1:8096). Choosing A or B is a one-line change in deploy/docker-compose.yml plus, for B, the NixOS change on .13.

---

6. Deferred, with owners

Item	Owner	Unblocked by
Deploy the real site (DoD 2)	this issue, woken by AUD-132	Phase 2 producing bin/server + dist/ from make build
Header + CSP proof (DoD 5)	this issue, after the Caddy block is applied	Sam applies §4
A1 over the public URL (DoD 6)	this issue, after 2 and 5	—
Caddy block applied and other hostnames confirmed (DoD 7)	Sam	§4
Bind-address decision (§5)	Sam	A or B above