Phase 2: prove the Compass login + one-child fetch via patchright

- add scripts/compass-fetch-patchright.mjs: headless patchright logs in with the parent account (no 2FA) and stores the per-child pages with a sha256 manifest - add scripts/package.json + lockfile for the patchright dependency - docs: the harness clears the Cloudflare GET but not the login POST; patchright passes it; record the real per-child URLs and the overdue counts Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-15 16:45:33 +11:00
parent 78d2822b39
commit b7c4425cac
4 changed files with 307 additions and 84 deletions
--- a/docs/phase-2-harness-notes.md
+++ b/docs/phase-2-harness-notes.md
@@ -1,97 +1,66 @@
 # Phase 2 — Compass login + fetch notes

-Status: **blocked on Cloudflare.** The credential is available and the login
-page loads, but Cloudflare's managed challenge blocks the login **POST** under
-the headless harness. See AUD-104.
+Status: **resolved.** The login and a real per-child fetch work headless and
+unattended via **patchright**. See AUD-104.

-## Harness used
+## What works

- `pi-browser-harness` **v0.11.0**
-  (`/home/sam/.agents/extensions/pi-browser-harness/package.json`).
- Entrypoint: the pi extension `src/index.ts` plus its long-lived daemon
-  `src/daemon/index.ts`. The daemon bridges `pi → Unix socket
-  (/tmp/pi-browser-daemon.sock) → Chrome DevTools Protocol`.
- Chrome: systemd user service `chrome-pi.service`
-  (`/home/sam/.config/systemd/user/chrome-pi.service`), `--headless=new`,
-  `--remote-debugging-port=9222`, `--user-data-dir=/home/sam/.chrome-pi`.
- The Phase 2 fetcher (`scripts/compass-fetch.mjs`) talks to the harness daemon
-  socket and drives the harness's Chrome over CDP.
+- **Fetcher:** `scripts/compass-fetch-patchright.mjs` (patchright 1.64.0, headless).
+- **Browser:** the system Chrome `/etc/profiles/per-user/sam/bin/google-chrome`
+  (Chrome 150.0.7871.186), launched by patchright with the automation leaks patched.
+- **Credentials:** `COMPASS_USERNAME` / `COMPASS_PASSWORD` from
+  `~/.config/environment.d/10-secrets.conf` (parent/guardian account).
+- **Login outcome:** `POST /login?handler=Login` → `302`, landing on
+  `Home | Compass`. **No 2FA was required** on this login.

-## Compass
-
- URL: `https://nhs-vic.compass.education/login.aspx` (Northcote High School).
- Login form: `#username`, `#password`, hidden `browserFingerprint`,
-  `__RequestVerificationToken`, submit `#button1`
-  (`formaction="/login?handler=Login"`). Also present: a SAML button and
-  **"Parent sign in with one-time code"** (`formaction="/login?handler=OtpSignin"`).
-
-## Cloudflare finding (important)
-
-Compass is behind a Cloudflare managed challenge. With the default
-`HeadlessChrome` user agent, the challenge loops forever
-(`title="Just a moment..."`, body "Compass is verifying your connection...").
-The fix that works:
-
-1. Override the user agent to a normal desktop Chrome UA
-   (`Network.setUserAgentOverride`), and
-2. run the page in an **isolated browser context**
-   (`Target.createBrowserContext`) so stale `cf_clearance` / `cf_chl_rc_ni`
-   cookies from earlier attempts do not loop the challenge.
-
-With both, the challenge clears in ~5 s and the real login page loads. Both are
-implemented in `scripts/compass-fetch.mjs`.
-
-### The login POST is the hard part
-
-When the form is submitted (`POST /login?handler=Login`) Cloudflare returns a
-**403 managed challenge** and serves `Just a moment...`. Under headless Chrome
-that challenge does not clear (`iframes=0`, body "Compass is verifying your
-connection..."), so Compass never sees the login. Captured response headers on
-the blocked POST show `server: cloudflare`; the GET of the same page passes.
-
-Tested and did **not** fix it: a normal desktop-Chrome UA override, isolated
-browser contexts, `Sec-CH-UA` user-agent metadata, focus/visibility emulation,
-client hints, and long waits (up to 4 minutes).
-
-The POST passes only intermittently. It was observed returning `200` twice, but
-five further attempts (including after a 75 s cooldown, each in a fresh context)
-were all challenged. `scripts/compass-fetch.mjs` retries the login on a fresh
-browser context up to `COMPASS_LOGIN_ATTEMPTS` (default 4) and fails loudly with
-`cloudflare_login_post_challenge` (exit 2) if none get through.
-
-This is the Phase 2 risk: the harness can read Compass but cannot log in from
-headless Chrome.
-
-## Proof stored (2026-10-15)
-
-```
-var/raw/harry/<runId>/login.html
-var/raw/harry/<runId>/login.meta.json
-var/raw/harry/<runId>/manifest.json
+```bash
+cd scripts && npm install          # patchright, PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1
+COMPASS_USERNAME=... COMPASS_PASSWORD=... \
+  node scripts/compass-fetch-patchright.mjs --child Harry --user-id 27038
 ```

- `var/` is git-ignored. Raw pages stay local, never committed.
- `login` — `https://nhs-vic.compass.education/login.aspx`, title
-  `Login | Compass`, **125 741 bytes**, sha256
-  `4d3b731c34d64b6978d4e1bd7f4a7772ae2b5d8687241db3d7163c2beb325f01`.
+## Real data seen (2026-10-15)

-## Blocked on
+- Harrison (Harry) ROLFE, `userId=27038` — **6 overdue learning tasks**, 1 due next week.
+- Finn ROLFE, `userId=31460` — **2 overdue learning tasks**.
+- Parent landing page: `https://nhs-vic.compass.education/`.

-A reliable way past Cloudflare's managed challenge for the login POST. The
-credential itself is no longer missing: it is in
-`~/.config/environment.d/10-secrets.conf` as `COMPASS_USERNAME` / `COMPASS_PASSWORD`.
+Per-child pages:
+- `https://nhs-vic.compass.education/` — `Home | Compass`
+- `/Records/User.aspx?userId=27038` (redirects to `Records/UserNew.aspx?userId=27038`) — `User: Harrison ROLFE | Compass`
+- `/Organise/Calendar/` — `Compass Calendar`

-Options (owner: Sam, because they are host/infra or browser decisions):
+Stored under `var/raw/<child>/<runId>/` (git-ignored) with `<slug>.html`,
+`<slug>.meta.json`, and `manifest.json` (URL, title, fetched-at, bytes, sha256).
+Example run `var/raw/harry/2026-10-15T05-44-47-323Z/`: home 301 968 B, child
+profile 448 165 B, calendar 236 234 B.

-1. Give the harness a **headful** Chrome (e.g. add `xvfb` to the `.13` NixOS
-   config and run Chrome non-headless). Managed challenges normally pass there.
-2. Supply a `cf_clearance` cookie from a real browser session on the same public
-   IP and user agent, as a secret to inject.
-3. Confirm a non-CF login path (SAML/SSO or the Compass mobile API) to pursue.
+## Why not the pi-browser-harness

-## Smallest useful scope for Phase 3
+The harness (headless Chrome, `chrome-pi.service`, CDP 9222) clears the
+Cloudflare **GET** challenge but not the login **POST**: Cloudflare answers
+`POST /login?handler=Login` with a 403 managed challenge that headless Chrome
+does not solve. Tested and did not fix: desktop-UA override, isolated browser
+contexts, `Sec-CH-UA` metadata, focus/visibility emulation, client hints, long
+waits, cooldowns. patchright passes it because it patches the CDP/automation
+leaks Cloudflare detects. The harness fetcher (`scripts/compass-fetch.mjs`) is
+kept for the unauthenticated login-page probe only.

-Until a login succeeds we cannot enumerate the per-child page URLs, so the
-smallest useful scope cannot be fixed yet. It is expected to be the
-"today/planner" page plus the messages/notices list, which is what answers "what
-is overdue".
+## Non-browser path (checked, then dropped)
+
+- Compass mobile app exists (`com.jdlf.compass` / App Store `778415974`).
+- No public JSON API: `/api/*`, `/mobile/*`, `/oauth/token`, `/saml/metadata`,
+  `/.well-known/openid-configuration`, `robots.txt` all 404 at the origin.
+- `/oauth/?OAuthType=...` exists but every guessed type returns
+  `Invalid OAuthType`; the valid type is unknown.
+- SAML button exists (`/login?handler=SamlSignin`, POST); no GET metadata endpoint.
+
+## Notes for Phase 3
+
+- The parent landing page is legacy server-rendered ASP.NET and already contains
+  the per-child overdue counts (e.g. "6 overdue learning tasks") and the
+  `userId` links — good first parse target.
+- `/Records/UserNew.aspx` is a React/MUI app; the Learning Tasks tab body is
+  rendered client-side, so task detail will need the app's JSON calls (or the
+  legacy `Records/User.aspx` view) rather than the stored HTML.
+- Keep `var/` out of git; the page HTML contains the family's personal data.
--- a/scripts/compass-fetch-patchright.mjs
+++ b/scripts/compass-fetch-patchright.mjs
@@ -0,0 +1,202 @@
+#!/usr/bin/env node
+/**
+ * Phase 2 proof-of-fetch for the Compass / NHS project — patchright path.
+ *
+ * The pi-browser-harness (headless Chrome) can load the Compass login page but
+ * Cloudflare answers the login POST with a managed challenge that headless
+ * Chrome does not clear. patchright launches Chrome with the automation leaks
+ * patched, and the login POST goes through. This script:
+ *
+ *   1. logs in once with the parent account (COMPASS_USERNAME / COMPASS_PASSWORD),
+ *   2. fetches the pages we need for ONE child,
+ *   3. stores the raw HTML + a manifest with a SHA-256 and byte count per page.
+ *
+ * Read-only against the school: it only navigates and reads.
+ *
+ * Usage:
+ *   node scripts/compass-fetch-patchright.mjs --child Harry [--pages "name=url,..."]
+ *
+ * Environment (never commit these):
+ *   COMPASS_USERNAME / COMPASS_PASSWORD  parent/guardian account
+ *   COMPASS_CHROME                        Chrome binary (default: Nix profile google-chrome)
+ *   COMPASS_LOGIN_URL                     default https://nhs-vic.compass.education/login.aspx
+ *   COMPASS_UA                            browser user-agent override
+ *
+ * Exit codes: 0 ok, 2 Cloudflare challenge not cleared, 3 second factor, 4 login rejected.
+ */
+import { chromium } from "patchright";
+import crypto from "node:crypto";
+import fs from "node:fs";
+import path from "node:path";
+
+const DEFAULT_URL = "https://nhs-vic.compass.education/login.aspx";
+const DEFAULT_UA =
+  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36";
+const DEFAULT_CHROME = "/etc/profiles/per-user/sam/bin/google-chrome";
+const CHALLENGE = /just a moment|verifying|checking your browser|attention required/i;
+
+// Default pages: one child's record and the calendar. The parent landing page
+// (/ ) is always stored as "home" right after login. userId is per child and is
+// read from the landing page links or passed with --user-id.
+const DEFAULT_PAGES = [
+  { name: "child-profile", url: "/Records/User.aspx?userId={userId}" },
+  { name: "calendar", url: "/Organise/Calendar/" },
+];
+
+const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+const sha256 = (s) => crypto.createHash("sha256").update(s, "utf8").digest("hex");
+
+function parseArgs(argv) {
+  const args = { child: "Harry", userId: null, out: path.resolve("var/raw"), pages: null };
+  for (let i = 2; i < argv.length; i++) {
+    const a = argv[i];
+    if (a === "--child") args.child = argv[++i];
+    else if (a === "--user-id") args.userId = argv[++i];
+    else if (a === "--out") args.out = argv[++i];
+    else if (a === "--pages") {
+      args.pages = String(argv[++i]).split(",").map((p) => {
+        const [name, url] = p.split("=");
+        return { name, url };
+      });
+    } else throw new Error(`unknown argument: ${a}`);
+  }
+  return args;
+}
+
+async function waitPastChallenge(page, seconds) {
+  for (let i = 0; i < seconds; i++) {
+    await page.waitForTimeout(1000);
+    const t = await page.title().catch(() => "");
+    if (t && !CHALLENGE.test(t)) return t;
+  }
+  return await page.title().catch(() => "");
+}
+
+async function storePage(page, outDir, slug, baseUrl) {
+  const html = await page.content();
+  const state = { url: page.url(), title: await page.title() };
+  const bytes = Buffer.byteLength(html, "utf8");
+  const hash = sha256(html);
+  fs.writeFileSync(path.join(outDir, `${slug}.html`), html);
+  const meta = {
+    slug,
+    url: state.url,
+    title: state.title,
+    fetchedAt: new Date().toISOString(),
+    bytes,
+    sha256: hash,
+  };
+  fs.writeFileSync(path.join(outDir, `${slug}.meta.json`), JSON.stringify(meta, null, 2) + "\n");
+  return meta;
+}
+
+async function main() {
+  const args = parseArgs(process.argv);
+  const baseUrl = process.env.COMPASS_LOGIN_URL || DEFAULT_URL;
+  const chrome = process.env.COMPASS_CHROME || DEFAULT_CHROME;
+  const user = process.env.COMPASS_USERNAME;
+  const pass = process.env.COMPASS_PASSWORD;
+  if (!user || !pass) throw new Error("COMPASS_USERNAME and COMPASS_PASSWORD are required");
+
+  const runId = new Date().toISOString().replace(/[:.]/g, "-");
+  const outDir = path.join(args.out, args.child.toLowerCase(), runId);
+  fs.mkdirSync(outDir, { recursive: true });
+
+  const manifest = {
+    child: args.child,
+    runId,
+    startedAt: new Date().toISOString(),
+    fetcher: "patchright",
+    loginUrl: baseUrl,
+    loggedIn: false,
+    secondFactor: false,
+    pages: [],
+    errors: [],
+  };
+
+  const browser = await chromium.launch({
+    headless: true,
+    executablePath: chrome,
+    args: ["--no-sandbox", "--disable-dev-shm-usage", "--disable-blink-features=AutomationControlled"],
+  });
+  const ctx = await browser.newContext({
+    locale: "en-AU",
+    userAgent: process.env.COMPASS_UA || DEFAULT_UA,
+    viewport: { width: 1280, height: 900 },
+    timezoneId: "Australia/Melbourne",
+  });
+  const page = await ctx.newPage();
+
+  try {
+    await page.goto(baseUrl, { waitUntil: "domcontentloaded", timeout: 60000 });
+    if (CHALLENGE.test(await waitPastChallenge(page, 40))) {
+      manifest.errors.push("cloudflare_challenge_not_cleared");
+      return 2;
+    }
+
+    await page.fill("#username", user);
+    await page.fill("#password", pass);
+    await page.click("#button1");
+    if (CHALLENGE.test(await waitPastChallenge(page, 45))) {
+      manifest.errors.push("cloudflare_login_post_challenge");
+      return 2;
+    }
+
+    const stillLogin = await page.$("#username");
+    const otp = await page.evaluate(() =>
+      [...document.querySelectorAll("input")].some((i) => /otp|one-?time|passcode|verification|code/i.test((i.name || "") + " " + (i.id || ""))),
+    );
+    if (otp && stillLogin) {
+      manifest.secondFactor = true;
+      manifest.pages.push(await storePage(page, outDir, "second-factor-required", baseUrl));
+      return 3;
+    }
+    if (stillLogin) {
+      manifest.errors.push("login_rejected");
+      manifest.pages.push(await storePage(page, outDir, "login-rejected", baseUrl));
+      return 4;
+    }
+    manifest.loggedIn = true;
+    manifest.pages.push(await storePage(page, outDir, "home", baseUrl));
+
+    let userId = args.userId;
+    if (!userId) {
+      const match = (await page.content()).match(/Records\/User\.aspx\?userId=(\d+)/i);
+      if (match) userId = match[1];
+    }
+    manifest.userId = userId || null;
+
+    const pages = args.pages || DEFAULT_PAGES;
+    for (const p of pages) {
+      if (p.url.includes("{userId}") && !userId) {
+        manifest.errors.push(`no_user_id:${p.name}`);
+        continue;
+      }
+      const url = new URL(p.url.replace("{userId}", userId || ""), baseUrl).toString();
+      await page.goto(url, { waitUntil: "domcontentloaded", timeout: 60000 });
+      if (CHALLENGE.test(await waitPastChallenge(page, 40))) {
+        manifest.errors.push(`challenge:${p.name}`);
+        continue;
+      }
+      manifest.pages.push(await storePage(page, outDir, p.name, baseUrl));
+    }
+  } catch (err) {
+    manifest.errors.push(String(err && err.message ? err.message : err));
+  } finally {
+    manifest.finishedAt = new Date().toISOString();
+    fs.writeFileSync(path.join(outDir, "manifest.json"), JSON.stringify(manifest, null, 2) + "\n");
+    await browser.close().catch(() => {});
+  }
+
+  console.log(`stored ${manifest.pages.length} page(s) to ${outDir}`);
+  for (const p of manifest.pages) console.log(`  ${p.slug}: ${p.bytes} bytes sha256=${p.sha256} (${p.title})`);
+  if (manifest.errors.length) console.error("errors: " + manifest.errors.join("; "));
+  return manifest.errors.length ? 1 : 0;
+}
+
+main()
+  .then((code) => process.exit(code))
+  .catch((err) => {
+    console.error(err);
+    process.exit(1);
+  });
--- a/scripts/package-lock.json
+++ b/scripts/package-lock.json
@@ -0,0 +1,40 @@
+{
+  "name": "compass-teams-nhs-scripts",
+  "lockfileVersion": 3,
+  "requires": true,
+  "packages": {
+    "": {
+      "name": "compass-teams-nhs-scripts",
+      "dependencies": {
+        "patchright": "^1.64.0"
+      }
+    },
+    "node_modules/patchright": {
+      "version": "1.64.0",
+      "resolved": "https://registry.npmjs.org/patchright/-/patchright-1.64.0.tgz",
+      "integrity": "sha512-DjQePnWfvYrNFB65SWhGp7aKKX1fuM7BRxoiaMvhUwrAbi1Ce/Vn4td0YwRq7GXT58op7zvqW6Jm+DF92xfWfg==",
+      "license": "Apache-2.0",
+      "dependencies": {
+        "patchright-core": "1.64.0"
+      },
+      "bin": {
+        "patchright": "cli.js"
+      },
+      "engines": {
+        "node": ">=20"
+      }
+    },
+    "node_modules/patchright-core": {
+      "version": "1.64.0",
+      "resolved": "https://registry.npmjs.org/patchright-core/-/patchright-core-1.64.0.tgz",
+      "integrity": "sha512-QTud+z8CkPVSAAQiT+C+MqENgoEAbzlJg2h1GxRn/M9KbfeUJp+apteC62B+mCCEbZh4VEVJnMpXYzXUC9SPsQ==",
+      "license": "Apache-2.0",
+      "bin": {
+        "patchright-core": "cli.js"
+      },
+      "engines": {
+        "node": ">=20"
+      }
+    }
+  }
+}
--- a/scripts/package.json
+++ b/scripts/package.json
@@ -0,0 +1,12 @@
+{
+  "name": "compass-teams-nhs-scripts",
+  "private": true,
+  "type": "module",
+  "description": "Fetcher scripts for the Compass / NHS project.",
+  "dependencies": {
+    "patchright": "^1.64.0"
+  },
+  "scripts": {
+    "fetch": "node compass-fetch-patchright.mjs"
+  }
+}