sam/obsidian-sorter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(outline): add AI agent context set (Outline layer)

Browse Source 
Add docs/outline/00-index.md plus system overview, config/env, code map,
and agent runbook in Simplified Technical English. Link the set from the
README. Completes the project-docs Outline layer.
This commit is contained in:
Sam Rolfe
2026-10-13 12:37:28 +11:00
parent eaa7761437
commit 63ed336dfa
6 changed files with 369 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
README.md
View File

@@ -43,6 +43,11 @@ flowchart LR
*Interactive Architecture Map:* [maps.lab.audasmedia.com.au/obsidian-sorter/docs/obsidian-sorter-map.html](https://maps.lab.audasmedia.com.au/obsidian-sorter/docs/obsidian-sorter-map.html) *Interactive Architecture Map:* [maps.lab.audasmedia.com.au/obsidian-sorter/docs/obsidian-sorter-map.html](https://maps.lab.audasmedia.com.au/obsidian-sorter/docs/obsidian-sorter-map.html)
## 📚 Documentation
- **AI agent context (Outline index)**: [`docs/outline/00-index.md`](docs/outline/00-index.md) — inject one file instead of searching the repo.
- **Handoff notes**: [`docs/HANDOFF.md`](docs/HANDOFF.md)
--- ---
## The taxonomy (simplified 2026-09-24) ## The taxonomy (simplified 2026-09-24)

39
docs/outline/00-index.md Normal file
View File

@@ -0,0 +1,39 @@
# obsidian-sorter — AI Document Index & Context
> **Target**: AI Agents
> **Writing Style**: ASD-STE100 Simplified Technical English (`ste-writing`). Brief, terse, active voice.
## 📚 Document Catalog (Inject Specific Document)
Read this index to inject the exact file you need. Do not search all files.
| Document Title | Path | Scope / Description |
| :--- | :--- | :--- |
| **System Overview** | `docs/outline/01-system-overview.md` | Explains project purpose, main architecture, and active services. |
| **Configuration & Env** | `docs/outline/02-config-and-env.md` | Lists environment variables, config paths, log paths, and secrets. |
| **Code Structure Map** | `docs/outline/03-code-map.md` | Shows folder structure, core entry points, and primary dependencies. |
| **AI Agent Runbook** | `docs/outline/04-agent-runbook.md` | Gives step-by-step commands to build, test, and restart agent work. |
---
## 🎯 System Summary
`obsidian-sorter` files notes into an Obsidian vault. A note enters `010 inbox/`.
A watcher proposes a destination. The owner approves the proposal. The writer
moves the note and updates the Table of Contents. Nothing moves without approval.
The code uses the Python standard library only. Hosts need no pip.
## ⚙️ Core Configuration
- **Env File**: `~/.config/environment.d/*.conf`, or `~/.config/obsidian-sorter/secrets.conf`
- **Config Directory**: `~/obsidian-sorter/` (deployed code)
- **Active Ports**: None. The sorter opens no listening port. It calls `api.typesafe.ai` over outbound HTTPS.
## 🔄 Agent Restart Steps
1. Read the Document Catalog above. Inject only required files.
2. Check git status: `git status`.
3. Check recent commits: `git log -n 5 --oneline`.
4. Run validation test: `python3 tests/run_tests.py --no-jev`.
5. Restart the watcher: `systemctl --user restart obsidian-sorter`.

77
docs/outline/01-system-overview.md Normal file
View File

@@ -0,0 +1,77 @@
# obsidian-sorter — System Overview
> **Target**: AI Agents — ASD-STE100 Simplified Technical English.
## Purpose
The sorter files notes into a personal Obsidian vault. The owner drops a note
into `010 inbox/`. The sorter decides the destination. The owner approves. The
sorter then moves the note and updates the index. The owner stays in control of
every move.
## Architecture
The sorter uses two classification tiers. Tier 1 uses deterministic rules. Tier
2 uses a probabilistic model (TypeSafe JEV). Tier 2 runs only when Tier 1 is
unsure.
The system has one rule for types: **the type mirrors the folder**. Nothing is
inferred from content. The destination folder decides the type.
| Folder | Type |
| :--- | :--- |
| `200 projects/` | `project` |
| `300 areas/` | `area` |
| `400 Personal Family/` | `family` |
| `100 Table of Contents/` | `index` |
| `000 daily/` | `daily` |
| `500 archive/` | `archive` |
| `Clippings/` | `clipping` |
| `010 inbox/` | `note` (unsorted placeholder only) |
Retired types (never reintroduce): `subject`, `domain`, `tool`, `service`,
`container`, `machine`, `storage`, `backup-system`, `network-component`,
`website`, `person`.
## Flow
1. A note lands in `010 inbox/` with `type: note`.
2. The watcher polls every 10 seconds. It classifies the note.
3. The watcher writes the proposal into the note frontmatter.
4. The owner reviews the note in Obsidian and ticks the `approved` checkbox.
5. The watcher files the note. It moves the file, strips admin fields, and
inserts the Table of Contents entry.
Admin fields written during review: `proposed_folder`, `proposed_toc`,
`proposed_tags`, `confidence`, `approved`, `proposed_for`. The writer removes
all admin fields after filing.
## Active Services
| Service | Host | Details |
| :--- | :--- | :--- |
| `obsidian-sorter` (systemd user unit) | `.27` | Polls the inbox every 10 seconds. Enabled and lingering. |
| Command wrapper | `.27` | `~/bin/obsidian-sort` |
The unit runs `~/bin/obsidian-sort watch --interval 10`. It writes to
`~/.local/state/obsidian-sorter.log`. It restarts on failure after 30 seconds.
## Key Behaviour Rules
- Dry run is the default. The `--apply` flag is required to write anything.
- Notes named `Untitled*` never move. The plan is blocked.
- The writer never overwrites an existing destination.
- The owner's folder choice always wins. The sorter never re-decides a folder
that already has a value.
- The owner's own tags win. A note tagged `project` or `family` stays in that
bucket.
- The writer preserves `tags`, `aliases`, `id`, and `created`.
- The writer never rewrites the note body.
- JEV is probabilistic. Folder accuracy is about 74%. Tests assert the bucket
strictly. Tests assert the sub-folder only with `--strict-folders`.
## Related Artifacts
- Gitea repository: `https://gitea.lab.audasmedia.com.au/sam/obsidian-sorter`
- Architecture map: `https://maps.lab.audasmedia.com.au/obsidian-sorter/docs/obsidian-sorter-map.html`
- Handoff notes: `docs/HANDOFF.md`

61
docs/outline/02-config-and-env.md Normal file
View File

@@ -0,0 +1,61 @@
# obsidian-sorter — Configuration & Environment
> **Target**: AI Agents — ASD-STE100 Simplified Technical English.
## Environment Variables
| Variable | Required | Purpose |
| :--- | :--- | :--- |
| `TYPESAFE_API_KEY` | Only for JEV | Authenticates calls to TypeSafe JEV. The rules tier needs no key. |
The sorter reads `TYPESAFE_API_KEY` in this order:
1. The process environment.
2. `~/.config/environment.d/*.conf`.
3. `~/.config/obsidian-sorter/secrets.conf`.
The sorter accepts `KEY=value` and `export KEY=value` lines. It strips quotes.
**Warning**: Non-login SSH shells do not load `environment.d`. Set the key or
use a login shell for manual JEV runs.
## Config Paths
| Path | Purpose |
| :--- | :--- |
| `~/obsidian/` | Vault root (default `--vault`). |
| `~/obsidian/010 inbox/` | Inbox folder (default `--inbox`). |
| `~/obsidian/010 inbox/_sort log.md` | Sorter activity log. |
| `~/.local/state/obsidian-sorter.log` | Watcher service log. |
| `~/.config/systemd/user/obsidian-sorter.service` | Watcher unit file. |
| `~/bin/obsidian-sort` | Command wrapper. Sets vault and inbox. |
The command-line default for `--inbox` is `inbox`. The `~/bin/obsidian-sort`
wrapper always passes `010 inbox`.
## Network
The sorter opens no listening port. It makes one outbound HTTPS call:
- `POST https://api.typesafe.ai/v1/systemone` (TypeSafe JEV).
Default request timeout is 90 seconds. Default call cap is 500 per run.
## Tunable Defaults
| Flag | Default | Meaning |
| :--- | :--- | :--- |
| `--threshold` | `0.45` | Rule confidence below this value escalates to JEV. |
| `--interval` | `5.0` (CLI), `10` (service) | Seconds between watcher passes. |
| `--max-calls` | `500` | Maximum JEV calls per run. |
| `--timeout` | `90.0` | JEV request timeout in seconds. |
## Secrets
Do not commit secrets. The API key lives only in the config files above. The
repository ignores `*.log`, `test-vault/`, `__pycache__/`, and `.venv/`.
## Cost
JEV costs about `$0.0001` per note. The rules tier is free. Use `--no-jev` to
run without API cost.

69
docs/outline/03-code-map.md Normal file
View File

@@ -0,0 +1,69 @@
# obsidian-sorter — Code Structure Map
> **Target**: AI Agents — ASD-STE100 Simplified Technical English.
## Dependencies
The sorter uses the Python standard library only. It requires Python 3.10 or
later. It uses no PyYAML, no HTTP library, and no JEV SDK. Deploy with `rsync`.
Hosts need no pip.
## Layout
| Path | Purpose |
| :--- | :--- |
| `sorter/__main__.py` | Module entry point. Calls `cli.main`. |
| `sorter/cli.py` | Argument parser and all commands. |
| `sorter/models.py` | Data types. Folder-to-type map. `Note`, `Proposal`. |
| `sorter/vault.py` | Vault scanning and index introspection. |
| `sorter/frontmatter.py` | Minimal YAML frontmatter reader and writer. |
| `sorter/classify.py` | Tier 1 deterministic classifier. |
| `sorter/jev.py` | TypeSafe JEV client. Cost tracking. |
| `sorter/tier2.py` | Tier 2 semantic classifier. Hybrid logic. |
| `sorter/writer.py` | Plans, frontmatter merge, TOC insert, apply. |
| `sorter/watch.py` | Inbox watcher. Maintains proposals. |
| `tests/run_tests.py` | Test suite. Runs against an isolated vault copy. |
| `tools/normalize_tags.py` | One-off tag cleanup tool. |
| `docs/` | Diagrams, handoff notes, and this Outline set. |
## Entry Points
1. **Command wrapper**: `~/bin/obsidian-sort`. It changes to `~/obsidian-sorter`,
then runs `python3 -m sorter` with the vault and inbox set.
2. **Module**: `python3 -m sorter --vault <path> --inbox <name> <command>`.
3. **Systemd unit**: `ExecStart=~/bin/obsidian-sort watch --interval 10`.
## Commands
Read-only commands:
| Command | Purpose |
| :--- | :--- |
| `scan` | Report vault shape. |
| `propose <file>` | Classify one note. |
| `inbox` | Classify every inbox note. |
| `validate` | Replay the classifier against known-good notes. |
| `benchmark` | Measure accuracy and cost of the hybrid classifier. |
Write commands:
| Command | Purpose |
| :--- | :--- |
| `sort <file>` | Plan, then optionally apply, sorting for one note. |
| `sort-inbox` | Plan, then optionally apply, sorting for the whole inbox. |
| `watch` | Watch the inbox and maintain proposals. |
## Core Data Flow
`vault.py` scans notes into `Note` objects. `classify.py` picks a folder with
rules. `tier2.py` escalates to `jev.py` when needed. `writer.py` builds a
`Proposal` and a `Plan`, then applies it. `watch.py` connects the classifier to
the file system on a 10-second poll.
## Where to Fix Things
- **Misfiled notes**: edit `FOLDER_NOTES` in `sorter/tier2.py`. This teaches the
model the meaning of each folder.
- **Rule behaviour**: edit `sorter/classify.py`.
- **Type or folder rules**: edit `sorter/models.py`.
- **Filing safety**: edit `sorter/writer.py`.

118
docs/outline/04-agent-runbook.md Normal file
View File

@@ -0,0 +1,118 @@
# obsidian-sorter — AI Agent Runbook
> **Target**: AI Agents — ASD-STE100 Simplified Technical English.
## Prerequisites
- Python 3.10 or later. No pip packages.
- `TYPESAFE_API_KEY` for the JEV tier. The rules tier needs no key.
- On `.27`, the code lives in `~/obsidian-sorter/`.
## Step 1 — Inspect Before You Act
Always run read-only commands first. Never write on the first pass.
```bash
cd ~/obsidian-sorter
# Vault shape: counts and types
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" scan
# Classify one note with the rules tier (free)
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
propose "010 inbox/My Note.md"
# Classify one note with JEV
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
propose "010 inbox/My Note.md" --jev
```
## Step 2 — Plan, Then Apply
`sort` and `sort-inbox` are dry runs by default. Add `--apply` to write. The
plan lists every change. Read the plan before you apply it.
```bash
# Show the plan
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
sort "010 inbox/My Note.md" --jev
# Apply one note
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
sort "010 inbox/My Note.md" --jev --apply
# Apply the whole inbox without a prompt (for agents)
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
sort-inbox --jev --apply --yes
```
## Step 3 — Run the Tests
The tests run against an isolated vault copy. They never touch the live vault.
```bash
cd ~/obsidian-sorter
python3 tests/run_tests.py # full suite (uses JEV, small cost)
python3 tests/run_tests.py --no-jev # mechanics only, no API cost
python3 tests/run_tests.py --keep # keep the temp vault for inspection
python3 tests/run_tests.py --strict-folders # assert sub-folders (may flake)
```
Validation command for agents: `python3 tests/run_tests.py --no-jev`.
## Step 4 — Refresh the Test Mirror (from `.13`)
```bash
rsync -a --delete \
--exclude='.git' --exclude='.trash' --exclude='.venv' \
--exclude='graphify-out*' --exclude='.obsidian' \
sam@192.168.20.27:~/obsidian/ ./test-vault/
```
## Step 5 — Manage the Watcher (on `.27`)
```bash
systemctl --user status obsidian-sorter # is it running?
systemctl --user restart obsidian-sorter # restart after a code change
systemctl --user stop obsidian-sorter # pause it
systemctl --user start obsidian-sorter # resume it
journalctl --user -u obsidian-sorter -f # live output
```
Log files:
- Service log: `~/.local/state/obsidian-sorter.log`
- Sort log: `~/obsidian/010 inbox/_sort log.md`
## Step 6 — Deploy Code to `.27`
```bash
rsync -a --delete --exclude='test-vault' --exclude='__pycache__' \
--exclude='.git' --exclude='*.pyc' ./ sam@192.168.20.27:~/obsidian-sorter/
systemctl --user restart obsidian-sorter # on .27
```
## Restart Checklist
1. Read this runbook and the Document Catalog in `00-index.md`.
2. Check git status: `git status`.
3. Check recent commits: `git log -n 5 --oneline`.
4. Run validation: `python3 tests/run_tests.py --no-jev`.
5. Restart the service: `systemctl --user restart obsidian-sorter`.
6. Confirm it runs: `systemctl --user status obsidian-sorter`.
## Safety Rules for Agents
- Never write to the live vault without `--apply`.
- Never edit a note body. Edit only frontmatter when the owner asks.
- Never move a note named `Untitled*`.
- Never overwrite an existing destination.
- Never reintroduce retired types.
- Never commit secrets.
## Current Gaps
- Phase D (MCP tool wrapper) is pending.
- Phase E (Pi skill + distribution) is pending.
- The test suite has no automated watcher test. Test it manually.