Phase 2: harden Compass fetcher for the Cloudflare login-POST challenge

- retry the login on a fresh browser context (stale cf_chl_rc_ni cookies loop
  the next challenge) and fail loudly with cloudflare_login_post_challenge
- detect the Compass login-rejection message and second-factor prompts
- docs: record that Cloudflare's managed challenge blocks the headless login
  POST (GET passes), what was tried, and the unblock options

Co-Authored-By: Paperclip <noreply@paperclip.ing>

docs/phase-2-harness-notes.md

@@ -1,8 +1,8 @@
 # Phase 2 — Compass login + fetch notes
 
-Status: **partially proven.** The browser path to Compass works; the login and
+Status: **blocked on Cloudflare.** The credential is available and the login
-per-child fetch are blocked on a Compass credential that is not available to the
+page loads, but Cloudflare's managed challenge blocks the login **POST** under
-agent. See AUD-104.
+the headless harness. See AUD-104.
 
 ## Harness used
 
@@ -41,6 +41,27 @@ The fix that works:
 With both, the challenge clears in ~5 s and the real login page loads. Both are
 implemented in `scripts/compass-fetch.mjs`.
 
+### The login POST is the hard part
+
+When the form is submitted (`POST /login?handler=Login`) Cloudflare returns a
+**403 managed challenge** and serves `Just a moment...`. Under headless Chrome
+that challenge does not clear (`iframes=0`, body "Compass is verifying your
+connection..."), so Compass never sees the login. Captured response headers on
+the blocked POST show `server: cloudflare`; the GET of the same page passes.
+
+Tested and did **not** fix it: a normal desktop-Chrome UA override, isolated
+browser contexts, `Sec-CH-UA` user-agent metadata, focus/visibility emulation,
+client hints, and long waits (up to 4 minutes).
+
+The POST passes only intermittently. It was observed returning `200` twice, but
+five further attempts (including after a 75 s cooldown, each in a fresh context)
+were all challenged. `scripts/compass-fetch.mjs` retries the login on a fresh
+browser context up to `COMPASS_LOGIN_ATTEMPTS` (default 4) and fails loudly with
+`cloudflare_login_post_challenge` (exit 2) if none get through.
+
+This is the Phase 2 risk: the harness can read Compass but cannot log in from
+headless Chrome.
+
 ## Proof stored (2026-10-15)
 
 ```
@@ -56,17 +77,21 @@ var/raw/harry/<runId>/manifest.json
 
 ## Blocked on
 
-A Compass account credential (not `HeadlessChrome`-related, not a Cloudflare
+A reliable way past Cloudflare's managed challenge for the login POST. The
-issue). Nothing matching `COMPASS*`, `SCHOOL*`, `NORTHCOTE*`, `HARRY*`, `FINN*`,
+credential itself is no longer missing: it is in
-or `TEAMS*` exists in `~/.config/environment.d/10-secrets.conf` or the paperclip
+`~/.config/environment.d/10-secrets.conf` as `COMPASS_USERNAME` / `COMPASS_PASSWORD`.
-secrets available to the agent. The account can be a parent/guardian account or
+
-per-child; confirmation of whether Compass enforces a one-time code on a new
+Options (owner: Sam, because they are host/infra or browser decisions):
-device is also needed.
+
+1. Give the harness a **headful** Chrome (e.g. add `xvfb` to the `.13` NixOS
+   config and run Chrome non-headless). Managed challenges normally pass there.
+2. Supply a `cf_clearance` cookie from a real browser session on the same public
+   IP and user agent, as a secret to inject.
+3. Confirm a non-CF login path (SAML/SSO or the Compass mobile API) to pursue.
 
 ## Smallest useful scope for Phase 3
 
-The login page exposes a username/password form and a parent one-time-code
+Until a login succeeds we cannot enumerate the per-child page URLs, so the
-option. Until a login succeeds we cannot enumerate the per-child page URLs, so
+smallest useful scope cannot be fixed yet. It is expected to be the
-the smallest useful scope cannot be fixed yet. It is expected to be the
 "today/planner" page plus the messages/notices list, which is what answers "what
 is overdue".

scripts/compass-fetch.mjs

@@ -40,6 +40,7 @@ const DEFAULT_URL = "https://nhs-vic.compass.education/login.aspx";
 const DEFAULT_UA =
   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36";
 const DAEMON_SOCKET = process.env.COMPASS_DAEMON_SOCKET || "/tmp/pi-browser-daemon.sock";
+const LOGIN_ATTEMPTS = Number(process.env.COMPASS_LOGIN_ATTEMPTS || 4);
 // Cloudflare's managed challenge loops forever under the default HeadlessChrome
 // UA. A normal desktop Chrome UA clears it. See AUD-104.
 const CHALLENGE_TITLE = /just a moment|verifying|checking your browser|attention required/i;
@@ -217,9 +218,9 @@ async function clearChallenge(api, sessionId, url) {
   return state;
 }
 
-async function login(api, sessionId, username, password) {
+async function attemptLogin(api, sessionId, username, password) {
   const form = await evalJs(api, sessionId, "!!document.querySelector('#username')");
-  if (!form) throw new Error("login form (#username) not present");
+  if (!form) return { formMissing: true, challenged: false, errorText: null, state: await pageState(api, sessionId) };
   await evalJs(
     api,
     sessionId,
@@ -237,12 +238,17 @@ async function login(api, sessionId, username, password) {
     })()`,
   );
   await evalJs(api, sessionId, "document.querySelector('#button1').click(); true");
-  return waitFor(
+  // Cloudflare challenges the login POST intermittently; poll until it clears.
-    api,
+  const state = await waitFor(api, sessionId, (s) => !CHALLENGE_TITLE.test(s.title), { attempts: 60, everyMs: 1000 });
-    sessionId,
+  const challenged = CHALLENGE_TITLE.test(state.title);
-    (s) => !/login\.aspx/i.test(s.url) || /dashboard|home|planner|messages/i.test(s.url),
+  const errorText = challenged
-    { attempts: 60, everyMs: 1000 },
+    ? null
-  );
+    : await evalJs(
+        api,
+        sessionId,
+        "(() => { const m = (document.body ? document.body.innerText : '').match(/Sorry,[^\\n]*/i); return m ? m[0].trim() : null; })()",
+      );
+  return { challenged, errorText, state, formMissing: false };
 }
 
 async function detectSecondFactor(api, sessionId) {
@@ -250,11 +256,15 @@ async function detectSecondFactor(api, sessionId) {
     api,
     sessionId,
     `(() => {
+      const url = location.href.toLowerCase();
       const txt = (document.body ? document.body.innerText : '').toLowerCase();
+      const loginish = /login|otp|verif|one-?time|two-?factor/.test(url) ||
+        /verification code|one-?time code|enter the code|we sent|two-factor|2fa/.test(txt);
       const otpInput = [...document.querySelectorAll('input')].some(
         (i) => /otp|one-?time|passcode|verification|code/i.test((i.name||'') + ' ' + (i.id||'')),
       );
-      return otpInput || /verification code|one-?time code|enter the code|we sent|two-factor|2fa/.test(txt);
+      const hasPassword = !!document.querySelector('#password');
+      return otpInput && loginish && !hasPassword;
     })()`,
   );
 }
@@ -297,7 +307,15 @@ async function main() {
   fs.mkdirSync(outDir, { recursive: true });
 
   const api = await connectDaemon(`compass-fetch-${process.pid}`);
-  const page = await newPage(api, args.child);
+  let page = await newPage(api, args.child);
+  // A fresh browser context per login attempt: stale Cloudflare challenge
+  // cookies (cf_chl_rc_ni) from a blocked POST loop the next challenge.
+  const freshPage = async () => {
+    const stale = page.browserContextId;
+    page = await newPage(api, args.child);
+    try { await api.send("Target.disposeBrowserContext", { browserContextId: stale }); } catch {}
+    return page;
+  };
   const manifest = {
     child: args.child,
     runId,
@@ -311,18 +329,32 @@ async function main() {
   };
 
   try {
-    const loginState = await clearChallenge(api, page.sessionId, baseUrl);
-    if (!loginState) {
-      manifest.errors.push("cloudflare_challenge_not_cleared");
-      fs.writeFileSync(path.join(outDir, "manifest.json"), JSON.stringify(manifest, null, 2) + "\n");
-      return 2;
-    }
-
     if (args.login) {
       const username = process.env.COMPASS_USERNAME;
       const password = process.env.COMPASS_PASSWORD;
       if (!username || !password) throw new Error("--login requires COMPASS_USERNAME and COMPASS_PASSWORD");
-      const afterLogin = await login(api, page.sessionId, username, password);
+      // Cloudflare intermittently 403s the login POST even when the GET is fine.
+      // Retry on a clean context until the POST reaches Compass.
+      let attempt = null;
+      for (let n = 1; n <= LOGIN_ATTEMPTS; n++) {
+        if (!(await clearChallenge(api, page.sessionId, baseUrl))) {
+          manifest.errors.push("cloudflare_challenge_not_cleared");
+          fs.writeFileSync(path.join(outDir, "manifest.json"), JSON.stringify(manifest, null, 2) + "\n");
+          return 2;
+        }
+        attempt = await attemptLogin(api, page.sessionId, username, password);
+        if (!attempt.challenged) break;
+        console.error(`login POST blocked by Cloudflare (attempt ${n}/${LOGIN_ATTEMPTS}); retrying on a fresh context`);
+        if (n < LOGIN_ATTEMPTS) {
+          await freshPage();
+          await sleep(8000);
+        }
+      }
+      if (attempt.challenged) {
+        manifest.errors.push("cloudflare_login_post_challenge");
+        fs.writeFileSync(path.join(outDir, "manifest.json"), JSON.stringify(manifest, null, 2) + "\n");
+        return 2;
+      }
       if (await detectSecondFactor(api, page.sessionId)) {
         manifest.secondFactor = true;
         manifest.pages.push(await storePage(api, page.sessionId, outDir, "second-factor-required"));
@@ -330,13 +362,22 @@ async function main() {
         console.error("Second factor / one-time code required. STOP — do not guess.");
         return 3;
       }
-      manifest.loggedIn = true;
+      const stillLoginForm = await evalJs(api, page.sessionId, "!!document.querySelector('#username')");
-      manifest.pages.push(await storePage(api, page.sessionId, outDir, "post-login"));
+      if (attempt.errorText || stillLoginForm) {
-      if (!/dashboard|home|planner|messages/i.test(afterLogin.url) && /login\.aspx/i.test(afterLogin.url)) {
+        manifest.errors.push(`login_rejected: ${attempt.errorText || "login form still present"}`);
+        manifest.pages.push(await storePage(api, page.sessionId, outDir, "login-rejected"));
         fs.writeFileSync(path.join(outDir, "manifest.json"), JSON.stringify(manifest, null, 2) + "\n");
         return 4;
       }
+      manifest.loggedIn = true;
+      manifest.pages.push(await storePage(api, page.sessionId, outDir, "post-login"));
     } else {
+      const loginState = await clearChallenge(api, page.sessionId, baseUrl);
+      if (!loginState) {
+        manifest.errors.push("cloudflare_challenge_not_cleared");
+        fs.writeFileSync(path.join(outDir, "manifest.json"), JSON.stringify(manifest, null, 2) + "\n");
+        return 2;
+      }
       manifest.pages.push(await storePage(api, page.sessionId, outDir, "login"));
     }