diff --git a/deploy/DEPLOYMENT.md b/deploy/DEPLOYMENT.md index a72f9d5..489e45e 100644 --- a/deploy/DEPLOYMENT.md +++ b/deploy/DEPLOYMENT.md @@ -100,12 +100,15 @@ the API directly on the internal address. ```bash deploy/setup-n8n-scan-key.sh --apply # restricted SSH key for the trigger - deploy/import-n8n-morning-brief.sh --apply # import + publish + restart n8n + deploy/import-n8n-morning-brief.sh --apply # credential + workflow + publish + restart n8n ``` - The workflow has one Execute Command node: `ssh … 'scripts/morning-brief.sh'`. - It needs no `SCHOOL_NOTIFY_TOKEN` in n8n — the host script reads the project env - file. See `docs/phase-7a-n8n-schedule.md` for the design and the failure drill. + The workflow is Schedule Trigger -> SSH node -> a Code check that makes a + non-zero exit red. The SSH node authenticates with the restricted key and runs + `scripts/morning-brief.sh` on the host. (n8n 2.x disables the Execute Command + node by default, so a command node cannot activate.) It needs no + `SCHOOL_NOTIFY_TOKEN` in n8n — the host script reads the project env file. See + `docs/phase-7a-n8n-schedule.md` for the design and the failure drill. Notes: @@ -120,4 +123,6 @@ Notes: - The 06:00 schedule is the n8n workflow `Compass / NHS — 6:00am morning brief` (`deploy/n8n-morning-brief.json`), triggered over a restricted SSH key. It runs on the shared n8n service on `.13`, so its import, publish, and the one-off n8n - restart need approval first. + restart need approval first. n8n 2.x disables the Execute Command node by default, + so the workflow uses the SSH node plus a Code check that fails the run on a + non-zero exit. diff --git a/deploy/import-n8n-morning-brief.sh b/deploy/import-n8n-morning-brief.sh index 81ef1f9..d51cf85 100755 --- a/deploy/import-n8n-morning-brief.sh +++ b/deploy/import-n8n-morning-brief.sh @@ -2,11 +2,23 @@ # Phase 7a (AUD-115): import + publish the 6:00am workflow in n8n. # # n8n on .13 is a shared family service. Run this only after the change is -# approved. It imports one workflow, publishes it, restarts n8n once so the -# 06:00 schedule registers, and verifies the published state. +# approved. It: +# 1. imports one SSH credential built from the restricted trigger key, +# 2. imports one workflow (schedule -> SSH -> failure check), +# 3. publishes the workflow, +# 4. restarts n8n once so the 06:00 schedule registers, +# 5. verifies the workflow is active in the n8n log, not just in the database. # -# It does NOT run the workflow. The first real run is the next 06:00 trigger -# (or `docker exec n8n n8n execute --id=` by hand). +# It does NOT run the workflow. The first real run is the next 06:00 trigger, +# or `docker exec n8n n8n execute --id=schoolMorningBrf` by hand (real send). +# +# The SSH credential holds the private half of the restricted key. It is built +# into a mode-0600 temp file, copied into the container, imported, and removed. +# It is never written into this repository. +# +# Why not an Execute Command node? n8n 2.x disables `n8n-nodes-base.executeCommand` +# by default (`nodes.exclude`), so such a workflow cannot activate. The SSH node +# is enabled, and the restricted key keeps it to one command. # # Usage: # deploy/import-n8n-morning-brief.sh # dry run, print the plan @@ -16,12 +28,25 @@ # Environment overrides: # CONTAINER n8n container name (default n8n) # WORKFLOW workflow json path (default /deploy/n8n-morning-brief.json) +# KEY private key path (default ~/.ssh/id_ed25519_school_scan) +# SSH_HOST host the brief runs on (default 192.168.20.13) +# SSH_USER user on that host (default sam) set -euo pipefail REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" CONTAINER="${CONTAINER:-n8n}" WORKFLOW="${WORKFLOW:-$REPO_ROOT/deploy/n8n-morning-brief.json}" -IN_CONTAINER="/tmp/n8n-morning-brief.json" +KEY="${KEY:-$HOME/.ssh/id_ed25519_school_scan}" +SSH_HOST="${SSH_HOST:-192.168.20.13}" +SSH_USER="${SSH_USER:-sam}" + +WF_ID="schoolMorningBrf" +WF_NAME="Compass / NHS — 6:00am morning brief" +CRED_ID="schoolScanKey" +CRED_NAME="school-scan restricted key" + +IN_CONTAINER_WF="/tmp/n8n-morning-brief.json" +IN_CONTAINER_CRED="/tmp/n8n-school-scan-cred.json" APPLY=0 RESTART=1 @@ -29,59 +54,70 @@ for arg in "$@"; do case "$arg" in --apply) APPLY=1 ;; --no-restart) RESTART=0 ;; - -h|--help) sed -n '2,20p' "$0"; exit 0 ;; + -h|--help) sed -n '2,30p' "$0"; exit 0 ;; *) echo "unknown argument: $arg" >&2; exit 2 ;; esac done [[ -f "$WORKFLOW" ]] || { echo "workflow not found: $WORKFLOW" >&2; exit 1; } -# The workflow name is stable; match on a substring to survive em-dash changes. -WF_MATCH="${WF_MATCH:-6:00am morning brief}" - n8n_cli() { docker exec "$CONTAINER" n8n "$@"; } -wf_id() { - n8n_cli list:workflow 2>/dev/null \ - | awk -F'|' -v m="$WF_MATCH" 'index($2, m) { print $1; exit }' -} - echo "workflow file : $WORKFLOW" echo "n8n container : $CONTAINER" -echo "in-container : $IN_CONTAINER" +echo "key : $KEY" +echo "ssh target : $SSH_USER@$SSH_HOST" echo "restart n8n : $([[ "$RESTART" -eq 1 ]] && echo yes || echo no)" if [[ "$APPLY" -eq 0 ]]; then echo echo "DRY RUN — no change to n8n. Re-run with --apply." echo "steps:" - echo " 1. copy the JSON into the container" - echo " 2. n8n import:workflow" - echo " 3. n8n publish:workflow --id=" - echo " 4. $([[ "$RESTART" -eq 1 ]] && echo 'docker restart n8n (≈10s)' || echo 'skip restart')" - echo " 5. n8n list:workflow --active=true shows the workflow" + echo " 1. build the SSH credential from $KEY and import it (id $CRED_ID)" + echo " 2. copy the workflow into the container" + echo " 3. n8n import:workflow" + echo " 4. n8n publish:workflow --id=$WF_ID" + echo " 5. $([[ "$RESTART" -eq 1 ]] && echo 'docker restart n8n (≈10s)' || echo 'skip restart')" + echo " 6. confirm the n8n log shows the workflow activated" exit 0 fi +[[ -f "$KEY" ]] || { echo "private key not found: $KEY (run deploy/setup-n8n-scan-key.sh --apply)" >&2; exit 1; } + +CRED_TMP="$(mktemp)" +AUTH_TMP="$(mktemp)" +cleanup() { rm -f "$CRED_TMP" "$AUTH_TMP"; docker exec "$CONTAINER" rm -f "$IN_CONTAINER_CRED" "$IN_CONTAINER_WF" >/dev/null 2>&1 || true; } +trap cleanup EXIT + +# 1. Build the credential JSON without ever printing the key. +umask 077 +python3 - "$KEY" "$CRED_ID" "$CRED_NAME" "$SSH_HOST" "$SSH_USER" > "$CRED_TMP" <<'PY' +import json, sys, pathlib +key_path, cred_id, cred_name, host, user = sys.argv[1:6] +key = pathlib.Path(key_path).read_text() +json.dump([{ + "id": cred_id, + "name": cred_name, + "type": "sshPrivateKey", + "data": {"host": host, "port": 22, "username": user, "privateKey": key, "passphrase": ""}, +}], sys.stdout, indent=2) +PY + echo -echo "== 1. copy the workflow into the container ==" -docker exec -i "$CONTAINER" sh -c "cat > $IN_CONTAINER" < "$WORKFLOW" +echo "== 1. import the SSH credential ==" +docker exec -i "$CONTAINER" sh -c "cat > $IN_CONTAINER_CRED" < "$CRED_TMP" +n8n_cli import:credentials --input="$IN_CONTAINER_CRED" -echo "== 2. import ==" -n8n_cli import:workflow --input="$IN_CONTAINER" +echo "== 2. copy the workflow into the container ==" +docker exec -i "$CONTAINER" sh -c "cat > $IN_CONTAINER_WF" < "$WORKFLOW" -ID="$(wf_id)" -if [[ -z "$ID" ]]; then - echo "could not find the imported workflow by name '$WF_MATCH'" >&2 - n8n_cli list:workflow >&2 || true - exit 1 -fi -echo "workflow id: $ID" +echo "== 3. import ==" +n8n_cli import:workflow --input="$IN_CONTAINER_WF" -echo "== 3. publish ==" -n8n_cli publish:workflow --id="$ID" +echo "== 4. publish ==" +n8n_cli publish:workflow --id="$WF_ID" -echo "== 4. restart ==" +echo "== 5. restart ==" if [[ "$RESTART" -eq 1 ]]; then docker restart "$CONTAINER" >/dev/null echo "n8n restarted; waiting for the API to answer…" @@ -92,10 +128,16 @@ if [[ "$RESTART" -eq 1 ]]; then fi sleep 2 done + sleep 3 fi -echo "== 5. verify ==" -n8n_cli list:workflow --active=true -echo -echo "AUTHORIZED_KEY_HINT: the Execute Command node calls the host with" -echo " ~/.ssh/id_ed25519_school_scan — create it with deploy/setup-n8n-scan-key.sh --apply" +echo "== 6. verify activation in the log ==" +if docker logs "$CONTAINER" 2>&1 | grep -q "Activated workflow \"$WF_NAME\""; then + echo "activated: $WF_NAME" +else + echo "the n8n log does not show an activation for '$WF_NAME' yet" >&2 + docker logs "$CONTAINER" 2>&1 | grep -iE "schoolMorningBrf|$WF_NAME" | tail -5 >&2 || true + echo "check: docker logs $CONTAINER | grep -i 'did fail'" >&2 + exit 1 +fi +n8n_cli list:workflow --active=true | grep "$WF_ID" || true diff --git a/deploy/n8n-morning-brief.json b/deploy/n8n-morning-brief.json index 6022115..e573546 100644 --- a/deploy/n8n-morning-brief.json +++ b/deploy/n8n-morning-brief.json @@ -17,19 +17,50 @@ "name": "6:00am every day", "type": "n8n-nodes-base.scheduleTrigger", "typeVersion": 1.3, - "position": [200, 300] + "position": [200, 360] + }, + { + "parameters": {}, + "id": "c1a0a0a0-0000-4000-8000-000000000002", + "name": "Run now (manual)", + "type": "n8n-nodes-base.manualTrigger", + "typeVersion": 1, + "position": [200, 180], + "notes": "Optional: run the brief by hand from the n8n UI, or with `docker exec n8n n8n execute --id=schoolMorningBrf`. The schedule fires on its own; this trigger only does something when you press it." }, { "parameters": { - "executeOnce": true, - "command": "ssh -i /home/node/.ssh/id_ed25519_school_scan -o IdentitiesOnly=yes -o BatchMode=yes -o UserKnownHostsFile=/home/node/.ssh/known_hosts -o StrictHostKeyChecking=yes sam@192.168.20.13 'cd /home/sam/paseo/projects/compass_teams_nhs && ./scripts/morning-brief.sh'" + "authentication": "privateKey", + "resource": "command", + "operation": "execute", + "command": "cd /home/sam/paseo/projects/compass_teams_nhs && ./scripts/morning-brief.sh", + "cwd": "/" }, - "id": "c1a0a0a0-0000-4000-8000-000000000002", - "name": "Scan both children and send the briefs", - "type": "n8n-nodes-base.executeCommand", + "id": "c1a0a0a0-0000-4000-8000-000000000003", + "name": "Run the morning brief on the host", + "type": "n8n-nodes-base.ssh", "typeVersion": 1, - "position": [480, 300], - "notes": "One bounded host command. The scan needs node + patchright on .13, so it runs outside this container. scripts/morning-brief.sh scans both children, builds the per-boy and combined briefs and sends them through Apprise, then exits non-zero if the scan or a send fails, so this execution goes red (FR-12). The key is restricted in authorized_keys to exactly this command." + "position": [480, 360], + "notes": "The scan needs node + patchright on .13, so it runs outside this container. The credential is the restricted trigger key: the server's forced command runs exactly scripts/morning-brief.sh, whatever command is sent.", + "credentials": { + "sshPrivateKey": { + "id": "schoolScanKey", + "name": "school-scan restricted key" + } + } + }, + { + "parameters": { + "mode": "runOnceForAllItems", + "language": "javaScript", + "jsCode": "for (const item of $input.all()) {\n if (item.json.code !== 0) {\n throw new Error('morning-brief.sh failed with exit ' + item.json.code + ': ' + (item.json.stderr || ''));\n }\n}\nreturn $input.all();" + }, + "id": "c1a0a0a0-0000-4000-8000-000000000004", + "name": "Red execution when the brief fails", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [720, 360], + "notes": "The SSH node reports the remote exit code but does not fail on it. This node turns a non-zero exit (a failed scan or a failed send) into a red execution, so the schedule can never look green when the brief failed (FR-12)." } ], "connections": { @@ -37,7 +68,29 @@ "main": [ [ { - "node": "Scan both children and send the briefs", + "node": "Run the morning brief on the host", + "type": "main", + "index": 0 + } + ] + ] + }, + "Run now (manual)": { + "main": [ + [ + { + "node": "Run the morning brief on the host", + "type": "main", + "index": 0 + } + ] + ] + }, + "Run the morning brief on the host": { + "main": [ + [ + { + "node": "Red execution when the brief fails", "type": "main", "index": 0 } diff --git a/docs/phase-7a-n8n-schedule.md b/docs/phase-7a-n8n-schedule.md index c56b68c..0552310 100644 --- a/docs/phase-7a-n8n-schedule.md +++ b/docs/phase-7a-n8n-schedule.md @@ -5,40 +5,55 @@ combined briefs, the Apprise channels, and eight real deliveries. This phase mak the schedule real: one n8n workflow that runs every day at 06:00 and sends the briefs. -## What changed +## What the workflow is -`deploy/n8n-morning-brief.json` is now a two-node workflow: +`deploy/n8n-morning-brief.json` is a four-node workflow: ``` -Schedule Trigger "0 6 * * *" (Australia/Melbourne) - | - v -Execute Command ssh … sam@192.168.20.13 'scripts/morning-brief.sh' +Schedule Trigger "0 6 * * *" (Australia/Melbourne) ─┐ + ├─▶ SSH node ──▶ Code check +Manual Trigger "Run now (manual)" ───────────────────┘ (restricted key) ``` -The earlier draft had three nodes (`scan` via `ssh remote` → `POST /notify/morning`). -Two problems made that shape wrong here: +- **SSH node** — authenticates to `.13` as `sam` with the restricted trigger key and + runs the brief on the host. The scan needs node + patchright on `.13`, so it cannot + run inside the n8n container. +- **Code node** — the SSH node returns the remote exit code but does not fail on it. + The Code node throws when `code !== 0`, so a failed scan or send turns the n8n + execution red (FR-12). +- **Manual Trigger** — optional. It lets Sam run the brief by hand from the n8n UI, or + with `docker exec n8n n8n execute --id=schoolMorningBrf`. The schedule fires on its + own; the manual trigger only does something when pressed. -1. **`ssh remote` does not work.** The n8n container mounts `~/.ssh` read-only, and - `~/.ssh/config` is a Nix symlink into `/nix/store`, which does not exist inside - the container. The hostname `remote` does not resolve, and the container key is - not in `~/.ssh/authorized_keys` (the host alias fails too). -2. **A split scan/send node hides scan failures.** If the scan node fails, n8n stops - before the HTTP node, so `POST /notify/morning` never runs and the family gets no - brief at all — the opposite of FR-12. +`scripts/morning-brief.sh` is the bounded command. It scans both children, builds the +per-boy and combined briefs, sends them through Apprise, and exits non-zero when the +scan or a send fails. The brief always opens with the failure when the scan failed — +never "nothing due". -`scripts/morning-brief.sh` is already the bounded command that does both steps and -exits non-zero when the scan or a send fails. One Execute Command node is therefore -both simpler and more correct: a failed scan turns the n8n execution red **and** still -sends a brief that opens with the failure (`SCAN FAILED`), never "nothing due". +The run needs **no** `SCHOOL_NOTIFY_TOKEN` in n8n. The token stays in +`environment.d/school-service.env`, which the host script reads. -Because the single command runs on the host, n8n needs **no** `SCHOOL_NOTIFY_TOKEN` -and no HTTP node. The token stays in `environment.d/school-service.env` only. +## Why not an Execute Command node -## The SSH trigger key +The first design used one `n8n-nodes-base.executeCommand` node. **It cannot work on +this instance.** n8n 2.x disables that node by default for security +(`nodes.exclude = ['n8n-nodes-base.executeCommand', 'n8n-nodes-base.localFileTrigger']`). +An imported workflow with it is marked active in the database but never activates: -The n8n container mounts `/home/sam/.ssh` read-only at `/home/node/.ssh`. The trigger -uses a dedicated key with a forced command, so it can run exactly one thing: +``` +Activation of workflow "Compass / NHS — 6:00am morning brief" (schoolMorningBrf) did fail + with error: "Unrecognized node type: n8n-nodes-base.executeCommand" +``` + +The same error affects the pre-existing `Tree Info` workflow. The SSH node is enabled +and, with the restricted key, runs exactly one command — so it is both workable and +least-privilege. Enabling Execute Command instead would need `NODES_EXCLUDE=[]` in the +shared n8n service, which re-enables arbitrary shell execution for every workflow. + +## The SSH trigger key and credential + +The n8n container mounts `/home/sam/.ssh` at `/home/node/.ssh`. The trigger uses a +dedicated key with a forced command, so it can run exactly one thing: ``` restrict,command="cd /home/sam/paseo/projects/compass_teams_nhs && ./scripts/morning-brief.sh" ssh-ed25519 AAAA… @@ -48,27 +63,33 @@ restrict,command="cd /home/sam/paseo/projects/compass_teams_nhs && ./scripts/mor - `command=` makes the server ignore whatever command the client sends. - The private key lives at `/home/sam/.ssh/id_ed25519_school_scan` (mode 0600) and is readable only by `sam` and the n8n container (which already mounts `~/.ssh`). -- The key is not sam's personal key and is not used anywhere else. +- The key is not Sam's personal key and is not used anywhere else. -Create it with: +Create the key with: ```bash deploy/setup-n8n-scan-key.sh # dry run: print the plan deploy/setup-n8n-scan-key.sh --apply # create the key + the authorized_keys line ``` +The SSH node needs the private key in an n8n credential (`sshPrivateKey`, id +`schoolScanKey`). `deploy/import-n8n-morning-brief.sh` builds that credential from the +key file into a mode-0600 temp file, imports it, and removes it. The key is never +written into this repository. + ## Import and activate ```bash deploy/import-n8n-morning-brief.sh # dry run: print the plan -deploy/import-n8n-morning-brief.sh --apply # import, publish, restart n8n +deploy/import-n8n-morning-brief.sh --apply # credential + workflow, publish, restart n8n ``` -The script copies the JSON into the container with `docker exec -i … sh -c 'cat > …'` -(`docker cp` fails against the read-only `~/.ssh` mount), runs -`n8n import:workflow`, then `n8n publish:workflow --id=` (n8n 2.x replaced -`update:workflow --active`), restarts n8n once so the 06:00 schedule registers, and -prints `n8n list:workflow --active=true`. +The script imports the credential, copies the JSON into the container with +`docker exec -i … sh -c 'cat > …'` (`docker cp` fails against the `~/.ssh` mount), +runs `n8n import:workflow`, then `n8n publish:workflow --id=schoolMorningBrf` (n8n 2.x +replaced `update:workflow --active`), restarts n8n once so the 06:00 schedule +registers, and confirms the **n8n log** shows the workflow activated — the database's +active flag alone is not proof. The JSON carries a stable `id` (`schoolMorningBrf`). n8n 2.18.5 refuses an import whose workflow has no `id` (`SQLITE_CONSTRAINT: NOT NULL constraint failed: @@ -76,17 +97,26 @@ workflow_entity.id`), so the id is required, not cosmetic. Re-importing the same updates the same workflow instead of adding a second copy. `n8n publish:workflow` prints *"Changes will not take effect if n8n is running"*, which -is why the import script restarts n8n once. The full import → publish → active check -was rehearsed in a throwaway n8n container with a temporary data directory before this -plan was proposed; no shared instance was touched. +is why the import script restarts n8n once. The full import → activate → run and the +failure path were rehearsed in a throwaway n8n container with a temporary data +directory, using the same n8n 2.18.5 image, before the change was proposed. The shared +instance was not touched for the rehearsal. ## Verification ```bash -docker exec n8n n8n list:workflow --active=true # the workflow is published +docker logs n8n | grep 'Activated workflow "Compass / NHS' +docker exec n8n n8n list:workflow --active=true # schoolMorningBrf is listed docker exec compass-teams-nhs /app/school notify status ``` +Rehearsal evidence (throwaway container, same image): + +- activation: `Activated workflow "school ssh rehearsal"` — no `Unrecognized node type` +- success path (forced command echo): `status: success` +- failure path (forced command `exit 7`): `status: error`, CLI exit 1, message + `morning-brief.sh failed with exit 7` + A healthy morning looks like this in `notify status`: ``` @@ -100,26 +130,24 @@ A healthy morning looks like this in `notify status`: 8 combined Sam whatsapp ok 06:0x ``` -Failure drill (never leave an empty day): point the key's forced command at a command -that exits non-zero, trigger the workflow, and confirm the n8n execution is red and -the brief body starts with the scan failure. Restore the command afterwards. - ## Proving a run -The Schedule Trigger only fires at 06:00, so the schedule itself is proved by -`n8n list:workflow --active=true` plus the next morning's `notify status` rows. To -prove the full send path without waiting, run the published workflow once by hand: +The Schedule Trigger only fires at 06:00, so the schedule is proved by the activation +log line plus the next morning's `notify status` rows. To prove the full send path +without waiting, run the published workflow once by hand: ```bash -docker exec n8n n8n execute --id= +docker exec n8n n8n execute --id=schoolMorningBrf ``` This sends the real briefs, so run it only when a real delivery is wanted. `n8n -list:workflow` prints the id. Both checks are recorded in this issue's final comment. +execute` starts from the Manual Trigger; both trigger paths reach the same SSH node. ## Not touched - Compass / Teams: read-only. Nothing in this phase writes to the school. - Other n8n workflows: untouched. +- n8n's global node configuration: untouched. Execute Command stays disabled. - Secrets: no address, phone number, Apprise key, session secret, or notify token is - written into the workflow JSON, this document, or git. + written into the workflow JSON, this document, or git. The SSH private key lives in + an n8n credential and the host `authorized_keys`, never in the repository. diff --git a/scripts/morning-brief.sh b/scripts/morning-brief.sh index b202b7d..9f12e61 100755 --- a/scripts/morning-brief.sh +++ b/scripts/morning-brief.sh @@ -22,7 +22,7 @@ # scripts/morning-brief.sh --channel email # limit the channels # # Environment overrides: -# IMAGE image tag (default 0.5.0-phase7) +# IMAGE image tag (default 0.6.1-phase8) # VOLUME SQLite volume (default compass-teams-nhs_compass-teams-nhs-data) # ENV_FILE service env file (default environment.d/school-service.env) # SCHOOL_TIMEZONE family timezone (default Australia/Sydney) @@ -31,7 +31,7 @@ set -euo pipefail REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -IMAGE="${IMAGE:-compass-teams-nhs/school:0.5.0-phase7}" +IMAGE="${IMAGE:-compass-teams-nhs/school:0.6.1-phase8}" VOLUME="${VOLUME:-compass-teams-nhs_compass-teams-nhs-data}" ENV_FILE="${ENV_FILE:-$REPO_ROOT/environment.d/school-service.env}" SCHOOL_TIMEZONE="${SCHOOL_TIMEZONE:-Australia/Sydney}"