docs(outline): add AI agent context set (Outline layer)

Add docs/outline/00-index.md plus system overview, config/env, code map, and agent runbook in Simplified Technical English. Link the set from the README. Completes the project-docs Outline layer.
2026-10-13 12:37:28 +11:00
parent eaa7761437
commit 63ed336dfa
6 changed files with 369 additions and 0 deletions
--- a/docs/outline/00-index.md
+++ b/docs/outline/00-index.md
@@ -0,0 +1,39 @@
+# obsidian-sorter — AI Document Index & Context
+
+> **Target**: AI Agents
+> **Writing Style**: ASD-STE100 Simplified Technical English (`ste-writing`). Brief, terse, active voice.
+
+## 📚 Document Catalog (Inject Specific Document)
+
+Read this index to inject the exact file you need. Do not search all files.
+
+| Document Title | Path | Scope / Description |
+| :--- | :--- | :--- |
+| **System Overview** | `docs/outline/01-system-overview.md` | Explains project purpose, main architecture, and active services. |
+| **Configuration & Env** | `docs/outline/02-config-and-env.md` | Lists environment variables, config paths, log paths, and secrets. |
+| **Code Structure Map** | `docs/outline/03-code-map.md` | Shows folder structure, core entry points, and primary dependencies. |
+| **AI Agent Runbook** | `docs/outline/04-agent-runbook.md` | Gives step-by-step commands to build, test, and restart agent work. |
+
+---
+
+## 🎯 System Summary
+
+`obsidian-sorter` files notes into an Obsidian vault. A note enters `010 inbox/`.
+A watcher proposes a destination. The owner approves the proposal. The writer
+moves the note and updates the Table of Contents. Nothing moves without approval.
+
+The code uses the Python standard library only. Hosts need no pip.
+
+## ⚙️ Core Configuration
+
+- **Env File**: `~/.config/environment.d/*.conf`, or `~/.config/obsidian-sorter/secrets.conf`
+- **Config Directory**: `~/obsidian-sorter/` (deployed code)
+- **Active Ports**: None. The sorter opens no listening port. It calls `api.typesafe.ai` over outbound HTTPS.
+
+## 🔄 Agent Restart Steps
+
+1. Read the Document Catalog above. Inject only required files.
+2. Check git status: `git status`.
+3. Check recent commits: `git log -n 5 --oneline`.
+4. Run validation test: `python3 tests/run_tests.py --no-jev`.
+5. Restart the watcher: `systemctl --user restart obsidian-sorter`.
--- a/docs/outline/01-system-overview.md
+++ b/docs/outline/01-system-overview.md
@@ -0,0 +1,77 @@
+# obsidian-sorter — System Overview
+
+> **Target**: AI Agents — ASD-STE100 Simplified Technical English.
+
+## Purpose
+
+The sorter files notes into a personal Obsidian vault. The owner drops a note
+into `010 inbox/`. The sorter decides the destination. The owner approves. The
+sorter then moves the note and updates the index. The owner stays in control of
+every move.
+
+## Architecture
+
+The sorter uses two classification tiers. Tier 1 uses deterministic rules. Tier
+2 uses a probabilistic model (TypeSafe JEV). Tier 2 runs only when Tier 1 is
+unsure.
+
+The system has one rule for types: **the type mirrors the folder**. Nothing is
+inferred from content. The destination folder decides the type.
+
+| Folder | Type |
+| :--- | :--- |
+| `200 projects/` | `project` |
+| `300 areas/` | `area` |
+| `400 Personal Family/` | `family` |
+| `100 Table of Contents/` | `index` |
+| `000 daily/` | `daily` |
+| `500 archive/` | `archive` |
+| `Clippings/` | `clipping` |
+| `010 inbox/` | `note` (unsorted placeholder only) |
+
+Retired types (never reintroduce): `subject`, `domain`, `tool`, `service`,
+`container`, `machine`, `storage`, `backup-system`, `network-component`,
+`website`, `person`.
+
+## Flow
+
+1. A note lands in `010 inbox/` with `type: note`.
+2. The watcher polls every 10 seconds. It classifies the note.
+3. The watcher writes the proposal into the note frontmatter.
+4. The owner reviews the note in Obsidian and ticks the `approved` checkbox.
+5. The watcher files the note. It moves the file, strips admin fields, and
+   inserts the Table of Contents entry.
+
+Admin fields written during review: `proposed_folder`, `proposed_toc`,
+`proposed_tags`, `confidence`, `approved`, `proposed_for`. The writer removes
+all admin fields after filing.
+
+## Active Services
+
+| Service | Host | Details |
+| :--- | :--- | :--- |
+| `obsidian-sorter` (systemd user unit) | `.27` | Polls the inbox every 10 seconds. Enabled and lingering. |
+| Command wrapper | `.27` | `~/bin/obsidian-sort` |
+
+The unit runs `~/bin/obsidian-sort watch --interval 10`. It writes to
+`~/.local/state/obsidian-sorter.log`. It restarts on failure after 30 seconds.
+
+## Key Behaviour Rules
+
+- Dry run is the default. The `--apply` flag is required to write anything.
+- Notes named `Untitled*` never move. The plan is blocked.
+- The writer never overwrites an existing destination.
+- The owner's folder choice always wins. The sorter never re-decides a folder
+  that already has a value.
+- The owner's own tags win. A note tagged `project` or `family` stays in that
+  bucket.
+- The writer preserves `tags`, `aliases`, `id`, and `created`.
+- The writer never rewrites the note body.
+- JEV is probabilistic. Folder accuracy is about 74%. Tests assert the bucket
+  strictly. Tests assert the sub-folder only with `--strict-folders`.
+
+## Related Artifacts
+
+- Gitea repository: `https://gitea.lab.audasmedia.com.au/sam/obsidian-sorter`
+- Architecture map: `https://maps.lab.audasmedia.com.au/obsidian-sorter/docs/obsidian-sorter-map.html`
+- Handoff notes: `docs/HANDOFF.md`
--- a/docs/outline/02-config-and-env.md
+++ b/docs/outline/02-config-and-env.md
@@ -0,0 +1,61 @@
+# obsidian-sorter — Configuration & Environment
+
+> **Target**: AI Agents — ASD-STE100 Simplified Technical English.
+
+## Environment Variables
+
+| Variable | Required | Purpose |
+| :--- | :--- | :--- |
+| `TYPESAFE_API_KEY` | Only for JEV | Authenticates calls to TypeSafe JEV. The rules tier needs no key. |
+
+The sorter reads `TYPESAFE_API_KEY` in this order:
+
+1. The process environment.
+2. `~/.config/environment.d/*.conf`.
+3. `~/.config/obsidian-sorter/secrets.conf`.
+
+The sorter accepts `KEY=value` and `export KEY=value` lines. It strips quotes.
+
+**Warning**: Non-login SSH shells do not load `environment.d`. Set the key or
+use a login shell for manual JEV runs.
+
+## Config Paths
+
+| Path | Purpose |
+| :--- | :--- |
+| `~/obsidian/` | Vault root (default `--vault`). |
+| `~/obsidian/010 inbox/` | Inbox folder (default `--inbox`). |
+| `~/obsidian/010 inbox/_sort log.md` | Sorter activity log. |
+| `~/.local/state/obsidian-sorter.log` | Watcher service log. |
+| `~/.config/systemd/user/obsidian-sorter.service` | Watcher unit file. |
+| `~/bin/obsidian-sort` | Command wrapper. Sets vault and inbox. |
+
+The command-line default for `--inbox` is `inbox`. The `~/bin/obsidian-sort`
+wrapper always passes `010 inbox`.
+
+## Network
+
+The sorter opens no listening port. It makes one outbound HTTPS call:
+
+- `POST https://api.typesafe.ai/v1/systemone` (TypeSafe JEV).
+
+Default request timeout is 90 seconds. Default call cap is 500 per run.
+
+## Tunable Defaults
+
+| Flag | Default | Meaning |
+| :--- | :--- | :--- |
+| `--threshold` | `0.45` | Rule confidence below this value escalates to JEV. |
+| `--interval` | `5.0` (CLI), `10` (service) | Seconds between watcher passes. |
+| `--max-calls` | `500` | Maximum JEV calls per run. |
+| `--timeout` | `90.0` | JEV request timeout in seconds. |
+
+## Secrets
+
+Do not commit secrets. The API key lives only in the config files above. The
+repository ignores `*.log`, `test-vault/`, `__pycache__/`, and `.venv/`.
+
+## Cost
+
+JEV costs about `$0.0001` per note. The rules tier is free. Use `--no-jev` to
+run without API cost.
--- a/docs/outline/03-code-map.md
+++ b/docs/outline/03-code-map.md
@@ -0,0 +1,69 @@
+# obsidian-sorter — Code Structure Map
+
+> **Target**: AI Agents — ASD-STE100 Simplified Technical English.
+
+## Dependencies
+
+The sorter uses the Python standard library only. It requires Python 3.10 or
+later. It uses no PyYAML, no HTTP library, and no JEV SDK. Deploy with `rsync`.
+Hosts need no pip.
+
+## Layout
+
+| Path | Purpose |
+| :--- | :--- |
+| `sorter/__main__.py` | Module entry point. Calls `cli.main`. |
+| `sorter/cli.py` | Argument parser and all commands. |
+| `sorter/models.py` | Data types. Folder-to-type map. `Note`, `Proposal`. |
+| `sorter/vault.py` | Vault scanning and index introspection. |
+| `sorter/frontmatter.py` | Minimal YAML frontmatter reader and writer. |
+| `sorter/classify.py` | Tier 1 deterministic classifier. |
+| `sorter/jev.py` | TypeSafe JEV client. Cost tracking. |
+| `sorter/tier2.py` | Tier 2 semantic classifier. Hybrid logic. |
+| `sorter/writer.py` | Plans, frontmatter merge, TOC insert, apply. |
+| `sorter/watch.py` | Inbox watcher. Maintains proposals. |
+| `tests/run_tests.py` | Test suite. Runs against an isolated vault copy. |
+| `tools/normalize_tags.py` | One-off tag cleanup tool. |
+| `docs/` | Diagrams, handoff notes, and this Outline set. |
+
+## Entry Points
+
+1. **Command wrapper**: `~/bin/obsidian-sort`. It changes to `~/obsidian-sorter`,
+   then runs `python3 -m sorter` with the vault and inbox set.
+2. **Module**: `python3 -m sorter --vault <path> --inbox <name> <command>`.
+3. **Systemd unit**: `ExecStart=~/bin/obsidian-sort watch --interval 10`.
+
+## Commands
+
+Read-only commands:
+
+| Command | Purpose |
+| :--- | :--- |
+| `scan` | Report vault shape. |
+| `propose <file>` | Classify one note. |
+| `inbox` | Classify every inbox note. |
+| `validate` | Replay the classifier against known-good notes. |
+| `benchmark` | Measure accuracy and cost of the hybrid classifier. |
+
+Write commands:
+
+| Command | Purpose |
+| :--- | :--- |
+| `sort <file>` | Plan, then optionally apply, sorting for one note. |
+| `sort-inbox` | Plan, then optionally apply, sorting for the whole inbox. |
+| `watch` | Watch the inbox and maintain proposals. |
+
+## Core Data Flow
+
+`vault.py` scans notes into `Note` objects. `classify.py` picks a folder with
+rules. `tier2.py` escalates to `jev.py` when needed. `writer.py` builds a
+`Proposal` and a `Plan`, then applies it. `watch.py` connects the classifier to
+the file system on a 10-second poll.
+
+## Where to Fix Things
+
+- **Misfiled notes**: edit `FOLDER_NOTES` in `sorter/tier2.py`. This teaches the
+  model the meaning of each folder.
+- **Rule behaviour**: edit `sorter/classify.py`.
+- **Type or folder rules**: edit `sorter/models.py`.
+- **Filing safety**: edit `sorter/writer.py`.
--- a/docs/outline/04-agent-runbook.md
+++ b/docs/outline/04-agent-runbook.md
@@ -0,0 +1,118 @@
+# obsidian-sorter — AI Agent Runbook
+
+> **Target**: AI Agents — ASD-STE100 Simplified Technical English.
+
+## Prerequisites
+
+- Python 3.10 or later. No pip packages.
+- `TYPESAFE_API_KEY` for the JEV tier. The rules tier needs no key.
+- On `.27`, the code lives in `~/obsidian-sorter/`.
+
+## Step 1 — Inspect Before You Act
+
+Always run read-only commands first. Never write on the first pass.
+
+```bash
+cd ~/obsidian-sorter
+
+# Vault shape: counts and types
+python3 -m sorter --vault ~/obsidian --inbox "010 inbox" scan
+
+# Classify one note with the rules tier (free)
+python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
+    propose "010 inbox/My Note.md"
+
+# Classify one note with JEV
+python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
+    propose "010 inbox/My Note.md" --jev
+```
+
+## Step 2 — Plan, Then Apply
+
+`sort` and `sort-inbox` are dry runs by default. Add `--apply` to write. The
+plan lists every change. Read the plan before you apply it.
+
+```bash
+# Show the plan
+python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
+    sort "010 inbox/My Note.md" --jev
+
+# Apply one note
+python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
+    sort "010 inbox/My Note.md" --jev --apply
+
+# Apply the whole inbox without a prompt (for agents)
+python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
+    sort-inbox --jev --apply --yes
+```
+
+## Step 3 — Run the Tests
+
+The tests run against an isolated vault copy. They never touch the live vault.
+
+```bash
+cd ~/obsidian-sorter
+
+python3 tests/run_tests.py            # full suite (uses JEV, small cost)
+python3 tests/run_tests.py --no-jev   # mechanics only, no API cost
+python3 tests/run_tests.py --keep     # keep the temp vault for inspection
+python3 tests/run_tests.py --strict-folders   # assert sub-folders (may flake)
+```
+
+Validation command for agents: `python3 tests/run_tests.py --no-jev`.
+
+## Step 4 — Refresh the Test Mirror (from `.13`)
+
+```bash
+rsync -a --delete \
+  --exclude='.git' --exclude='.trash' --exclude='.venv' \
+  --exclude='graphify-out*' --exclude='.obsidian' \
+  sam@192.168.20.27:~/obsidian/ ./test-vault/
+```
+
+## Step 5 — Manage the Watcher (on `.27`)
+
+```bash
+systemctl --user status  obsidian-sorter    # is it running?
+systemctl --user restart obsidian-sorter    # restart after a code change
+systemctl --user stop    obsidian-sorter    # pause it
+systemctl --user start   obsidian-sorter    # resume it
+journalctl --user -u obsidian-sorter -f     # live output
+```
+
+Log files:
+
+- Service log: `~/.local/state/obsidian-sorter.log`
+- Sort log: `~/obsidian/010 inbox/_sort log.md`
+
+## Step 6 — Deploy Code to `.27`
+
+```bash
+rsync -a --delete --exclude='test-vault' --exclude='__pycache__' \
+  --exclude='.git' --exclude='*.pyc' ./ sam@192.168.20.27:~/obsidian-sorter/
+systemctl --user restart obsidian-sorter    # on .27
+```
+
+## Restart Checklist
+
+1. Read this runbook and the Document Catalog in `00-index.md`.
+2. Check git status: `git status`.
+3. Check recent commits: `git log -n 5 --oneline`.
+4. Run validation: `python3 tests/run_tests.py --no-jev`.
+5. Restart the service: `systemctl --user restart obsidian-sorter`.
+6. Confirm it runs: `systemctl --user status obsidian-sorter`.
+
+## Safety Rules for Agents
+
+- Never write to the live vault without `--apply`.
+- Never edit a note body. Edit only frontmatter when the owner asks.
+- Never move a note named `Untitled*`.
+- Never overwrite an existing destination.
+- Never reintroduce retired types.
+- Never commit secrets.
+
+## Current Gaps
+
+- Phase D (MCP tool wrapper) is pending.
+- Phase E (Pi skill + distribution) is pending.
+- The test suite has no automated watcher test. Test it manually.