Phase 2: prove the Compass login + one-child fetch via patchright
- add scripts/compass-fetch-patchright.mjs: headless patchright logs in with the parent account (no 2FA) and stores the per-child pages with a sha256 manifest - add scripts/package.json + lockfile for the patchright dependency - docs: the harness clears the Cloudflare GET but not the login POST; patchright passes it; record the real per-child URLs and the overdue counts Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Paperclip Builder
@@ -1,97 +1,66 @@
|
||||
# Phase 2 — Compass login + fetch notes
|
||||
|
||||
Status: **blocked on Cloudflare.** The credential is available and the login
|
||||
page loads, but Cloudflare's managed challenge blocks the login **POST** under
|
||||
the headless harness. See AUD-104.
|
||||
Status: **resolved.** The login and a real per-child fetch work headless and
|
||||
unattended via **patchright**. See AUD-104.
|
||||
|
||||
## Harness used
|
||||
## What works
|
||||
|
||||
- `pi-browser-harness` **v0.11.0**
|
||||
(`/home/sam/.agents/extensions/pi-browser-harness/package.json`).
|
||||
- Entrypoint: the pi extension `src/index.ts` plus its long-lived daemon
|
||||
`src/daemon/index.ts`. The daemon bridges `pi → Unix socket
|
||||
(/tmp/pi-browser-daemon.sock) → Chrome DevTools Protocol`.
|
||||
- Chrome: systemd user service `chrome-pi.service`
|
||||
(`/home/sam/.config/systemd/user/chrome-pi.service`), `--headless=new`,
|
||||
`--remote-debugging-port=9222`, `--user-data-dir=/home/sam/.chrome-pi`.
|
||||
- The Phase 2 fetcher (`scripts/compass-fetch.mjs`) talks to the harness daemon
|
||||
socket and drives the harness's Chrome over CDP.
|
||||
- **Fetcher:** `scripts/compass-fetch-patchright.mjs` (patchright 1.64.0, headless).
|
||||
- **Browser:** the system Chrome `/etc/profiles/per-user/sam/bin/google-chrome`
|
||||
(Chrome 150.0.7871.186), launched by patchright with the automation leaks patched.
|
||||
- **Credentials:** `COMPASS_USERNAME` / `COMPASS_PASSWORD` from
|
||||
`~/.config/environment.d/10-secrets.conf` (parent/guardian account).
|
||||
- **Login outcome:** `POST /login?handler=Login` → `302`, landing on
|
||||
`Home | Compass`. **No 2FA was required** on this login.
|
||||
|
||||
## Compass
|
||||
|
||||
- URL: `https://nhs-vic.compass.education/login.aspx` (Northcote High School).
|
||||
- Login form: `#username`, `#password`, hidden `browserFingerprint`,
|
||||
`__RequestVerificationToken`, submit `#button1`
|
||||
(`formaction="/login?handler=Login"`). Also present: a SAML button and
|
||||
**"Parent sign in with one-time code"** (`formaction="/login?handler=OtpSignin"`).
|
||||
|
||||
## Cloudflare finding (important)
|
||||
|
||||
Compass is behind a Cloudflare managed challenge. With the default
|
||||
`HeadlessChrome` user agent, the challenge loops forever
|
||||
(`title="Just a moment..."`, body "Compass is verifying your connection...").
|
||||
The fix that works:
|
||||
|
||||
1. Override the user agent to a normal desktop Chrome UA
|
||||
(`Network.setUserAgentOverride`), and
|
||||
2. run the page in an **isolated browser context**
|
||||
(`Target.createBrowserContext`) so stale `cf_clearance` / `cf_chl_rc_ni`
|
||||
cookies from earlier attempts do not loop the challenge.
|
||||
|
||||
With both, the challenge clears in ~5 s and the real login page loads. Both are
|
||||
implemented in `scripts/compass-fetch.mjs`.
|
||||
|
||||
### The login POST is the hard part
|
||||
|
||||
When the form is submitted (`POST /login?handler=Login`) Cloudflare returns a
|
||||
**403 managed challenge** and serves `Just a moment...`. Under headless Chrome
|
||||
that challenge does not clear (`iframes=0`, body "Compass is verifying your
|
||||
connection..."), so Compass never sees the login. Captured response headers on
|
||||
the blocked POST show `server: cloudflare`; the GET of the same page passes.
|
||||
|
||||
Tested and did **not** fix it: a normal desktop-Chrome UA override, isolated
|
||||
browser contexts, `Sec-CH-UA` user-agent metadata, focus/visibility emulation,
|
||||
client hints, and long waits (up to 4 minutes).
|
||||
|
||||
The POST passes only intermittently. It was observed returning `200` twice, but
|
||||
five further attempts (including after a 75 s cooldown, each in a fresh context)
|
||||
were all challenged. `scripts/compass-fetch.mjs` retries the login on a fresh
|
||||
browser context up to `COMPASS_LOGIN_ATTEMPTS` (default 4) and fails loudly with
|
||||
`cloudflare_login_post_challenge` (exit 2) if none get through.
|
||||
|
||||
This is the Phase 2 risk: the harness can read Compass but cannot log in from
|
||||
headless Chrome.
|
||||
|
||||
## Proof stored (2026-10-15)
|
||||
|
||||
```
|
||||
var/raw/harry/<runId>/login.html
|
||||
var/raw/harry/<runId>/login.meta.json
|
||||
var/raw/harry/<runId>/manifest.json
|
||||
```bash
|
||||
cd scripts && npm install # patchright, PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1
|
||||
COMPASS_USERNAME=... COMPASS_PASSWORD=... \
|
||||
node scripts/compass-fetch-patchright.mjs --child Harry --user-id 27038
|
||||
```
|
||||
|
||||
- `var/` is git-ignored. Raw pages stay local, never committed.
|
||||
- `login` — `https://nhs-vic.compass.education/login.aspx`, title
|
||||
`Login | Compass`, **125 741 bytes**, sha256
|
||||
`4d3b731c34d64b6978d4e1bd7f4a7772ae2b5d8687241db3d7163c2beb325f01`.
|
||||
## Real data seen (2026-10-15)
|
||||
|
||||
## Blocked on
|
||||
- Harrison (Harry) ROLFE, `userId=27038` — **6 overdue learning tasks**, 1 due next week.
|
||||
- Finn ROLFE, `userId=31460` — **2 overdue learning tasks**.
|
||||
- Parent landing page: `https://nhs-vic.compass.education/`.
|
||||
|
||||
A reliable way past Cloudflare's managed challenge for the login POST. The
|
||||
credential itself is no longer missing: it is in
|
||||
`~/.config/environment.d/10-secrets.conf` as `COMPASS_USERNAME` / `COMPASS_PASSWORD`.
|
||||
Per-child pages:
|
||||
- `https://nhs-vic.compass.education/` — `Home | Compass`
|
||||
- `/Records/User.aspx?userId=27038` (redirects to `Records/UserNew.aspx?userId=27038`) — `User: Harrison ROLFE | Compass`
|
||||
- `/Organise/Calendar/` — `Compass Calendar`
|
||||
|
||||
Options (owner: Sam, because they are host/infra or browser decisions):
|
||||
Stored under `var/raw/<child>/<runId>/` (git-ignored) with `<slug>.html`,
|
||||
`<slug>.meta.json`, and `manifest.json` (URL, title, fetched-at, bytes, sha256).
|
||||
Example run `var/raw/harry/2026-10-15T05-44-47-323Z/`: home 301 968 B, child
|
||||
profile 448 165 B, calendar 236 234 B.
|
||||
|
||||
1. Give the harness a **headful** Chrome (e.g. add `xvfb` to the `.13` NixOS
|
||||
config and run Chrome non-headless). Managed challenges normally pass there.
|
||||
2. Supply a `cf_clearance` cookie from a real browser session on the same public
|
||||
IP and user agent, as a secret to inject.
|
||||
3. Confirm a non-CF login path (SAML/SSO or the Compass mobile API) to pursue.
|
||||
## Why not the pi-browser-harness
|
||||
|
||||
## Smallest useful scope for Phase 3
|
||||
The harness (headless Chrome, `chrome-pi.service`, CDP 9222) clears the
|
||||
Cloudflare **GET** challenge but not the login **POST**: Cloudflare answers
|
||||
`POST /login?handler=Login` with a 403 managed challenge that headless Chrome
|
||||
does not solve. Tested and did not fix: desktop-UA override, isolated browser
|
||||
contexts, `Sec-CH-UA` metadata, focus/visibility emulation, client hints, long
|
||||
waits, cooldowns. patchright passes it because it patches the CDP/automation
|
||||
leaks Cloudflare detects. The harness fetcher (`scripts/compass-fetch.mjs`) is
|
||||
kept for the unauthenticated login-page probe only.
|
||||
|
||||
Until a login succeeds we cannot enumerate the per-child page URLs, so the
|
||||
smallest useful scope cannot be fixed yet. It is expected to be the
|
||||
"today/planner" page plus the messages/notices list, which is what answers "what
|
||||
is overdue".
|
||||
## Non-browser path (checked, then dropped)
|
||||
|
||||
- Compass mobile app exists (`com.jdlf.compass` / App Store `778415974`).
|
||||
- No public JSON API: `/api/*`, `/mobile/*`, `/oauth/token`, `/saml/metadata`,
|
||||
`/.well-known/openid-configuration`, `robots.txt` all 404 at the origin.
|
||||
- `/oauth/?OAuthType=...` exists but every guessed type returns
|
||||
`Invalid OAuthType`; the valid type is unknown.
|
||||
- SAML button exists (`/login?handler=SamlSignin`, POST); no GET metadata endpoint.
|
||||
|
||||
## Notes for Phase 3
|
||||
|
||||
- The parent landing page is legacy server-rendered ASP.NET and already contains
|
||||
the per-child overdue counts (e.g. "6 overdue learning tasks") and the
|
||||
`userId` links — good first parse target.
|
||||
- `/Records/UserNew.aspx` is a React/MUI app; the Learning Tasks tab body is
|
||||
rendered client-side, so task detail will need the app's JSON calls (or the
|
||||
legacy `Records/User.aspx` view) rather than the stored HTML.
|
||||
- Keep `var/` out of git; the page HTML contains the family's personal data.
|
||||
|
||||
Reference in New Issue
Block a user