obsidian-sorter: automated note filing for an Obsidian vault
Classifies notes from an inbox folder and files them, with a human approval step. Zero third-party dependencies. - type mirrors folder (200 projects / 300 areas / 400 Personal Family) - TypeSafe JEV for semantic classification (~$0.0001 per note) - watcher writes a proposal INTO the note; the owner ticks a checkbox - systemd user service on .27, 10s poll - 15/15 functional tests
This commit is contained in:
sam
130
docs/HANDOFF.md
Normal file
130
docs/HANDOFF.md
Normal file
@@ -0,0 +1,130 @@
|
||||
# HANDOFF — Obsidian Sorter
|
||||
|
||||
> Written 2026-09-24. Pick this up cold; everything you need is here.
|
||||
|
||||
## What this is
|
||||
|
||||
An automated system that files notes into Sam's Obsidian vault.
|
||||
|
||||
Notes go into `010 inbox/`. A **watcher** proposes a destination by writing it
|
||||
*into the note's frontmatter*. Sam ticks a checkbox in Obsidian to approve.
|
||||
The watcher then files it and updates the Table of Contents.
|
||||
|
||||
**Nothing is ever filed without approval** — deliberately, so Sam can edit a note
|
||||
before it moves.
|
||||
|
||||
## Where things live
|
||||
|
||||
| Thing | Location |
|
||||
|-------|----------|
|
||||
| Vault (live) | `.27` → `~/obsidian` — git-synced to Gitea `sam/obsidian-vault` |
|
||||
| Sorter code (dev) | `.13` → `~/paseo/projects/obsidian-sorter/` |
|
||||
| Sorter code (deployed) | `.27` → `~/obsidian-sorter/` |
|
||||
| Watcher service | `.27` → `systemctl --user obsidian-sorter` (enabled + lingering) |
|
||||
| Command | `.27` → `~/bin/obsidian-sort` |
|
||||
| Sort log | `.27` → `~/obsidian/010 inbox/_sort log.md` |
|
||||
| Service log | `.27` → `~/.local/state/obsidian-sorter.log` |
|
||||
| Design | `~/paseo/projects/obsidian-cleanup-tool/SORTER-PLAN.md` |
|
||||
|
||||
## THE TAXONOMY — final, do not "improve" it
|
||||
|
||||
**Type mirrors folder.** Nothing is inferred from content.
|
||||
|
||||
| Folder | Type |
|
||||
|--------|------|
|
||||
| `200 projects/` | `project` |
|
||||
| `300 areas/` | `area` |
|
||||
| `400 Personal Family/` | `family` |
|
||||
| `100 Table of Contents/` | `index` |
|
||||
| `000 daily/` | `daily` |
|
||||
| `500 archive/` | `archive` |
|
||||
| `Clippings/` | `clipping` |
|
||||
| `010 inbox/` | `note` (unsorted placeholder only) |
|
||||
|
||||
**RETIRED — never reintroduce:** `subject`, `domain`, `tool`, `service`,
|
||||
`container`, `machine`, `storage`, `backup-system`, `network-component`,
|
||||
`website`, `person`.
|
||||
|
||||
Sam's lifecycle model:
|
||||
```
|
||||
IDEA (not started) -> 200 projects/260 Ideas to Build Create Make
|
||||
BEING BUILT -> 200 projects/250 Electronics
|
||||
FINISHED -> 300 areas/...
|
||||
```
|
||||
|
||||
## The workflow
|
||||
|
||||
1. Note lands in `010 inbox/` with `type: note`
|
||||
2. Watcher (polls every 10s) classifies it and **writes the proposal into the note**:
|
||||
```yaml
|
||||
proposed_folder: 200 projects/250 Electronics
|
||||
proposed_toc: 100 Table of Contents/Projects.md → Ideas
|
||||
proposed_tags: [solar, 12v, bird, bath]
|
||||
confidence: 0.69
|
||||
approved: false # a checkbox in Obsidian
|
||||
```
|
||||
3. Sam ticks `approved` (or edits the proposal first)
|
||||
4. Watcher files it: move, set type from folder, merge tags, **strip admin
|
||||
fields**, update the TOC
|
||||
|
||||
## Implementation notes
|
||||
|
||||
- **Zero dependencies.** Custom YAML parser (`sorter/frontmatter.py`) — no PyYAML,
|
||||
because NixOS has no pip. HTTP via stdlib `urllib`.
|
||||
- **JEV** (TypeSafe, `POST api.typesafe.ai/v1/systemone`) does semantic
|
||||
classification — ~$0.0001/note. Key read from `~/.config/environment.d/*.conf`
|
||||
(not in the env for non-login SSH shells).
|
||||
- **Classifies by FOLDER, not type.** JEV picks the folder; the type follows.
|
||||
- **Sam's own tags win.** A note tagged `project`/`family` is constrained to that
|
||||
bucket — JEV cannot override it (`explicit_bucket_from_tags`).
|
||||
- **Folder meanings are taught explicitly** in `FOLDER_NOTES` (tier2.py), e.g.
|
||||
*"260 Ideas = UNSTARTED; 250 Electronics = ACTIVELY BEING BUILT"*.
|
||||
**This is the place to fix any future misfiling.**
|
||||
- JEV is **probabilistic** (~74% folder accuracy). Tests assert the *bucket*
|
||||
strictly, the sub-folder only with `--strict-folders`.
|
||||
- Files starting with `_` are ignored by the inbox scanner, so `_sort log.md`
|
||||
and `_review` files never get sorted.
|
||||
|
||||
## Safety guarantees (15/15 tests)
|
||||
|
||||
Dry run by default · `Untitled*` never moved · never overwrites an existing
|
||||
destination · frontmatter merged (user `tags`/`aliases`/`id`/`created` survive) ·
|
||||
body never rewritten · atomic writes · `.trash/` untouched · idempotent TOC.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
# on .27
|
||||
obsidian-sort # dry run — show what would happen
|
||||
obsidian-sort apply # file the notes (asks to confirm)
|
||||
obsidian-sort scan # vault summary
|
||||
systemctl --user status obsidian-sorter
|
||||
journalctl --user -u obsidian-sorter -f
|
||||
|
||||
# tests (on .13, against the mirror)
|
||||
cd ~/paseo/projects/obsidian-sorter
|
||||
python3 tests/run_tests.py
|
||||
python3 tests/run_tests.py --strict-folders # may flake (JEV variance)
|
||||
|
||||
# refresh the local mirror from the live vault
|
||||
rsync -a --delete --exclude='.git' --exclude='.trash' --exclude='.venv' \
|
||||
--exclude='graphify-out*' --exclude='.obsidian' \
|
||||
sam@192.168.20.27:~/obsidian/ ./test-vault/
|
||||
|
||||
# deploy code to .27
|
||||
rsync -a --delete --exclude='test-vault' --exclude='__pycache__' \
|
||||
--exclude='.git' --exclude='*.pyc' ./ sam@192.168.20.27:~/obsidian-sorter/
|
||||
```
|
||||
|
||||
## Tag vocabulary
|
||||
|
||||
Cleaned 2026-09-24: 214 → 195 unique tags; lowercased everything, merged
|
||||
`tools`→`tool`, split one malformed comma-joined tag into 9, fixed typos,
|
||||
removed all retired-type tags. Tool: `tools/normalize_tags.py`.
|
||||
|
||||
## Remaining work
|
||||
|
||||
- **Phase D** — MCP tool so Pi agents can call the sorter
|
||||
- **Phase E** — Pi skill ("sort my inbox") + distribute via Gitea to .13/.27/.51
|
||||
- Add a watcher test to the suite (currently only verified manually)
|
||||
- Sam is still testing with real notes — **no Pi integration until he's satisfied**
|
||||
Reference in New Issue
Block a user