diff --git a/README.md b/README.md index 9b1df5b..69cfe45 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,11 @@ flowchart LR *Interactive Architecture Map:* [maps.lab.audasmedia.com.au/obsidian-sorter/docs/obsidian-sorter-map.html](https://maps.lab.audasmedia.com.au/obsidian-sorter/docs/obsidian-sorter-map.html) +## 📚 Documentation + +- **AI agent context (Outline index)**: [`docs/outline/00-index.md`](docs/outline/00-index.md) — inject one file instead of searching the repo. +- **Handoff notes**: [`docs/HANDOFF.md`](docs/HANDOFF.md) + --- ## The taxonomy (simplified 2026-09-24) diff --git a/docs/outline/00-index.md b/docs/outline/00-index.md new file mode 100644 index 0000000..6009e04 --- /dev/null +++ b/docs/outline/00-index.md @@ -0,0 +1,39 @@ +# obsidian-sorter — AI Document Index & Context + +> **Target**: AI Agents +> **Writing Style**: ASD-STE100 Simplified Technical English (`ste-writing`). Brief, terse, active voice. + +## 📚 Document Catalog (Inject Specific Document) + +Read this index to inject the exact file you need. Do not search all files. + +| Document Title | Path | Scope / Description | +| :--- | :--- | :--- | +| **System Overview** | `docs/outline/01-system-overview.md` | Explains project purpose, main architecture, and active services. | +| **Configuration & Env** | `docs/outline/02-config-and-env.md` | Lists environment variables, config paths, log paths, and secrets. | +| **Code Structure Map** | `docs/outline/03-code-map.md` | Shows folder structure, core entry points, and primary dependencies. | +| **AI Agent Runbook** | `docs/outline/04-agent-runbook.md` | Gives step-by-step commands to build, test, and restart agent work. | + +--- + +## 🎯 System Summary + +`obsidian-sorter` files notes into an Obsidian vault. A note enters `010 inbox/`. +A watcher proposes a destination. The owner approves the proposal. The writer +moves the note and updates the Table of Contents. Nothing moves without approval. + +The code uses the Python standard library only. Hosts need no pip. + +## ⚙️ Core Configuration + +- **Env File**: `~/.config/environment.d/*.conf`, or `~/.config/obsidian-sorter/secrets.conf` +- **Config Directory**: `~/obsidian-sorter/` (deployed code) +- **Active Ports**: None. The sorter opens no listening port. It calls `api.typesafe.ai` over outbound HTTPS. + +## 🔄 Agent Restart Steps + +1. Read the Document Catalog above. Inject only required files. +2. Check git status: `git status`. +3. Check recent commits: `git log -n 5 --oneline`. +4. Run validation test: `python3 tests/run_tests.py --no-jev`. +5. Restart the watcher: `systemctl --user restart obsidian-sorter`. diff --git a/docs/outline/01-system-overview.md b/docs/outline/01-system-overview.md new file mode 100644 index 0000000..6695ee3 --- /dev/null +++ b/docs/outline/01-system-overview.md @@ -0,0 +1,77 @@ +# obsidian-sorter — System Overview + +> **Target**: AI Agents — ASD-STE100 Simplified Technical English. + +## Purpose + +The sorter files notes into a personal Obsidian vault. The owner drops a note +into `010 inbox/`. The sorter decides the destination. The owner approves. The +sorter then moves the note and updates the index. The owner stays in control of +every move. + +## Architecture + +The sorter uses two classification tiers. Tier 1 uses deterministic rules. Tier +2 uses a probabilistic model (TypeSafe JEV). Tier 2 runs only when Tier 1 is +unsure. + +The system has one rule for types: **the type mirrors the folder**. Nothing is +inferred from content. The destination folder decides the type. + +| Folder | Type | +| :--- | :--- | +| `200 projects/` | `project` | +| `300 areas/` | `area` | +| `400 Personal Family/` | `family` | +| `100 Table of Contents/` | `index` | +| `000 daily/` | `daily` | +| `500 archive/` | `archive` | +| `Clippings/` | `clipping` | +| `010 inbox/` | `note` (unsorted placeholder only) | + +Retired types (never reintroduce): `subject`, `domain`, `tool`, `service`, +`container`, `machine`, `storage`, `backup-system`, `network-component`, +`website`, `person`. + +## Flow + +1. A note lands in `010 inbox/` with `type: note`. +2. The watcher polls every 10 seconds. It classifies the note. +3. The watcher writes the proposal into the note frontmatter. +4. The owner reviews the note in Obsidian and ticks the `approved` checkbox. +5. The watcher files the note. It moves the file, strips admin fields, and + inserts the Table of Contents entry. + +Admin fields written during review: `proposed_folder`, `proposed_toc`, +`proposed_tags`, `confidence`, `approved`, `proposed_for`. The writer removes +all admin fields after filing. + +## Active Services + +| Service | Host | Details | +| :--- | :--- | :--- | +| `obsidian-sorter` (systemd user unit) | `.27` | Polls the inbox every 10 seconds. Enabled and lingering. | +| Command wrapper | `.27` | `~/bin/obsidian-sort` | + +The unit runs `~/bin/obsidian-sort watch --interval 10`. It writes to +`~/.local/state/obsidian-sorter.log`. It restarts on failure after 30 seconds. + +## Key Behaviour Rules + +- Dry run is the default. The `--apply` flag is required to write anything. +- Notes named `Untitled*` never move. The plan is blocked. +- The writer never overwrites an existing destination. +- The owner's folder choice always wins. The sorter never re-decides a folder + that already has a value. +- The owner's own tags win. A note tagged `project` or `family` stays in that + bucket. +- The writer preserves `tags`, `aliases`, `id`, and `created`. +- The writer never rewrites the note body. +- JEV is probabilistic. Folder accuracy is about 74%. Tests assert the bucket + strictly. Tests assert the sub-folder only with `--strict-folders`. + +## Related Artifacts + +- Gitea repository: `https://gitea.lab.audasmedia.com.au/sam/obsidian-sorter` +- Architecture map: `https://maps.lab.audasmedia.com.au/obsidian-sorter/docs/obsidian-sorter-map.html` +- Handoff notes: `docs/HANDOFF.md` diff --git a/docs/outline/02-config-and-env.md b/docs/outline/02-config-and-env.md new file mode 100644 index 0000000..6bdce18 --- /dev/null +++ b/docs/outline/02-config-and-env.md @@ -0,0 +1,61 @@ +# obsidian-sorter — Configuration & Environment + +> **Target**: AI Agents — ASD-STE100 Simplified Technical English. + +## Environment Variables + +| Variable | Required | Purpose | +| :--- | :--- | :--- | +| `TYPESAFE_API_KEY` | Only for JEV | Authenticates calls to TypeSafe JEV. The rules tier needs no key. | + +The sorter reads `TYPESAFE_API_KEY` in this order: + +1. The process environment. +2. `~/.config/environment.d/*.conf`. +3. `~/.config/obsidian-sorter/secrets.conf`. + +The sorter accepts `KEY=value` and `export KEY=value` lines. It strips quotes. + +**Warning**: Non-login SSH shells do not load `environment.d`. Set the key or +use a login shell for manual JEV runs. + +## Config Paths + +| Path | Purpose | +| :--- | :--- | +| `~/obsidian/` | Vault root (default `--vault`). | +| `~/obsidian/010 inbox/` | Inbox folder (default `--inbox`). | +| `~/obsidian/010 inbox/_sort log.md` | Sorter activity log. | +| `~/.local/state/obsidian-sorter.log` | Watcher service log. | +| `~/.config/systemd/user/obsidian-sorter.service` | Watcher unit file. | +| `~/bin/obsidian-sort` | Command wrapper. Sets vault and inbox. | + +The command-line default for `--inbox` is `inbox`. The `~/bin/obsidian-sort` +wrapper always passes `010 inbox`. + +## Network + +The sorter opens no listening port. It makes one outbound HTTPS call: + +- `POST https://api.typesafe.ai/v1/systemone` (TypeSafe JEV). + +Default request timeout is 90 seconds. Default call cap is 500 per run. + +## Tunable Defaults + +| Flag | Default | Meaning | +| :--- | :--- | :--- | +| `--threshold` | `0.45` | Rule confidence below this value escalates to JEV. | +| `--interval` | `5.0` (CLI), `10` (service) | Seconds between watcher passes. | +| `--max-calls` | `500` | Maximum JEV calls per run. | +| `--timeout` | `90.0` | JEV request timeout in seconds. | + +## Secrets + +Do not commit secrets. The API key lives only in the config files above. The +repository ignores `*.log`, `test-vault/`, `__pycache__/`, and `.venv/`. + +## Cost + +JEV costs about `$0.0001` per note. The rules tier is free. Use `--no-jev` to +run without API cost. diff --git a/docs/outline/03-code-map.md b/docs/outline/03-code-map.md new file mode 100644 index 0000000..feb2632 --- /dev/null +++ b/docs/outline/03-code-map.md @@ -0,0 +1,69 @@ +# obsidian-sorter — Code Structure Map + +> **Target**: AI Agents — ASD-STE100 Simplified Technical English. + +## Dependencies + +The sorter uses the Python standard library only. It requires Python 3.10 or +later. It uses no PyYAML, no HTTP library, and no JEV SDK. Deploy with `rsync`. +Hosts need no pip. + +## Layout + +| Path | Purpose | +| :--- | :--- | +| `sorter/__main__.py` | Module entry point. Calls `cli.main`. | +| `sorter/cli.py` | Argument parser and all commands. | +| `sorter/models.py` | Data types. Folder-to-type map. `Note`, `Proposal`. | +| `sorter/vault.py` | Vault scanning and index introspection. | +| `sorter/frontmatter.py` | Minimal YAML frontmatter reader and writer. | +| `sorter/classify.py` | Tier 1 deterministic classifier. | +| `sorter/jev.py` | TypeSafe JEV client. Cost tracking. | +| `sorter/tier2.py` | Tier 2 semantic classifier. Hybrid logic. | +| `sorter/writer.py` | Plans, frontmatter merge, TOC insert, apply. | +| `sorter/watch.py` | Inbox watcher. Maintains proposals. | +| `tests/run_tests.py` | Test suite. Runs against an isolated vault copy. | +| `tools/normalize_tags.py` | One-off tag cleanup tool. | +| `docs/` | Diagrams, handoff notes, and this Outline set. | + +## Entry Points + +1. **Command wrapper**: `~/bin/obsidian-sort`. It changes to `~/obsidian-sorter`, + then runs `python3 -m sorter` with the vault and inbox set. +2. **Module**: `python3 -m sorter --vault --inbox `. +3. **Systemd unit**: `ExecStart=~/bin/obsidian-sort watch --interval 10`. + +## Commands + +Read-only commands: + +| Command | Purpose | +| :--- | :--- | +| `scan` | Report vault shape. | +| `propose ` | Classify one note. | +| `inbox` | Classify every inbox note. | +| `validate` | Replay the classifier against known-good notes. | +| `benchmark` | Measure accuracy and cost of the hybrid classifier. | + +Write commands: + +| Command | Purpose | +| :--- | :--- | +| `sort ` | Plan, then optionally apply, sorting for one note. | +| `sort-inbox` | Plan, then optionally apply, sorting for the whole inbox. | +| `watch` | Watch the inbox and maintain proposals. | + +## Core Data Flow + +`vault.py` scans notes into `Note` objects. `classify.py` picks a folder with +rules. `tier2.py` escalates to `jev.py` when needed. `writer.py` builds a +`Proposal` and a `Plan`, then applies it. `watch.py` connects the classifier to +the file system on a 10-second poll. + +## Where to Fix Things + +- **Misfiled notes**: edit `FOLDER_NOTES` in `sorter/tier2.py`. This teaches the + model the meaning of each folder. +- **Rule behaviour**: edit `sorter/classify.py`. +- **Type or folder rules**: edit `sorter/models.py`. +- **Filing safety**: edit `sorter/writer.py`. diff --git a/docs/outline/04-agent-runbook.md b/docs/outline/04-agent-runbook.md new file mode 100644 index 0000000..03cf207 --- /dev/null +++ b/docs/outline/04-agent-runbook.md @@ -0,0 +1,118 @@ +# obsidian-sorter — AI Agent Runbook + +> **Target**: AI Agents — ASD-STE100 Simplified Technical English. + +## Prerequisites + +- Python 3.10 or later. No pip packages. +- `TYPESAFE_API_KEY` for the JEV tier. The rules tier needs no key. +- On `.27`, the code lives in `~/obsidian-sorter/`. + +## Step 1 — Inspect Before You Act + +Always run read-only commands first. Never write on the first pass. + +```bash +cd ~/obsidian-sorter + +# Vault shape: counts and types +python3 -m sorter --vault ~/obsidian --inbox "010 inbox" scan + +# Classify one note with the rules tier (free) +python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \ + propose "010 inbox/My Note.md" + +# Classify one note with JEV +python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \ + propose "010 inbox/My Note.md" --jev +``` + +## Step 2 — Plan, Then Apply + +`sort` and `sort-inbox` are dry runs by default. Add `--apply` to write. The +plan lists every change. Read the plan before you apply it. + +```bash +# Show the plan +python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \ + sort "010 inbox/My Note.md" --jev + +# Apply one note +python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \ + sort "010 inbox/My Note.md" --jev --apply + +# Apply the whole inbox without a prompt (for agents) +python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \ + sort-inbox --jev --apply --yes +``` + +## Step 3 — Run the Tests + +The tests run against an isolated vault copy. They never touch the live vault. + +```bash +cd ~/obsidian-sorter + +python3 tests/run_tests.py # full suite (uses JEV, small cost) +python3 tests/run_tests.py --no-jev # mechanics only, no API cost +python3 tests/run_tests.py --keep # keep the temp vault for inspection +python3 tests/run_tests.py --strict-folders # assert sub-folders (may flake) +``` + +Validation command for agents: `python3 tests/run_tests.py --no-jev`. + +## Step 4 — Refresh the Test Mirror (from `.13`) + +```bash +rsync -a --delete \ + --exclude='.git' --exclude='.trash' --exclude='.venv' \ + --exclude='graphify-out*' --exclude='.obsidian' \ + sam@192.168.20.27:~/obsidian/ ./test-vault/ +``` + +## Step 5 — Manage the Watcher (on `.27`) + +```bash +systemctl --user status obsidian-sorter # is it running? +systemctl --user restart obsidian-sorter # restart after a code change +systemctl --user stop obsidian-sorter # pause it +systemctl --user start obsidian-sorter # resume it +journalctl --user -u obsidian-sorter -f # live output +``` + +Log files: + +- Service log: `~/.local/state/obsidian-sorter.log` +- Sort log: `~/obsidian/010 inbox/_sort log.md` + +## Step 6 — Deploy Code to `.27` + +```bash +rsync -a --delete --exclude='test-vault' --exclude='__pycache__' \ + --exclude='.git' --exclude='*.pyc' ./ sam@192.168.20.27:~/obsidian-sorter/ +systemctl --user restart obsidian-sorter # on .27 +``` + +## Restart Checklist + +1. Read this runbook and the Document Catalog in `00-index.md`. +2. Check git status: `git status`. +3. Check recent commits: `git log -n 5 --oneline`. +4. Run validation: `python3 tests/run_tests.py --no-jev`. +5. Restart the service: `systemctl --user restart obsidian-sorter`. +6. Confirm it runs: `systemctl --user status obsidian-sorter`. + +## Safety Rules for Agents + +- Never write to the live vault without `--apply`. +- Never edit a note body. Edit only frontmatter when the owner asks. +- Never move a note named `Untitled*`. +- Never overwrite an existing destination. +- Never reintroduce retired types. +- Never commit secrets. + +## Current Gaps + +- Phase D (MCP tool wrapper) is pending. +- Phase E (Pi skill + distribution) is pending. +- The test suite has no automated watcher test. Test it manually.