obsidian-sorter: automated note filing for an Obsidian vault
Classifies notes from an inbox folder and files them, with a human approval step. Zero third-party dependencies. - type mirrors folder (200 projects / 300 areas / 400 Personal Family) - TypeSafe JEV for semantic classification (~$0.0001 per note) - watcher writes a proposal INTO the note; the owner ticks a checkbox - systemd user service on .27, 10s poll - 15/15 functional tests
This commit is contained in:
sam
254
README.md
Normal file
254
README.md
Normal file
@@ -0,0 +1,254 @@
|
||||
# obsidian-sorter
|
||||
|
||||
Automated note classification and filing for an Obsidian vault.
|
||||
|
||||
Notes enter through `010 inbox/`, the sorter decides where they belong, and —
|
||||
once approved — writes them into place and updates the Table of Contents.
|
||||
|
||||
**Design goal: no dependencies.** The vault uses simple, uniform YAML; the tool
|
||||
ships a small purpose-built parser so it deploys onto hosts without pip
|
||||
(NixOS `.13`, `.27`, `.51`).
|
||||
|
||||
---
|
||||
|
||||
## The taxonomy (simplified 2026-09-24)
|
||||
|
||||
**The type of a note mirrors the folder it lives in.** Nothing is guessed from
|
||||
content — the destination decides the type.
|
||||
|
||||
| Folder | Type | Meaning |
|
||||
|--------|------|---------|
|
||||
| `200 projects/` | `project` | Work in progress — something being built |
|
||||
| `300 areas/` | `area` | Ongoing interest with documentation and notes |
|
||||
| `400 Personal Family/` | `family` | Family / non-work |
|
||||
| `100 Table of Contents/` | `index` | Navigation notes only |
|
||||
| `000 daily/` | `daily` | Chronological daily notes |
|
||||
| `500 archive/` | `archive` | Inactive material |
|
||||
| `Clippings/` | `clipping` | Browser-extension captures |
|
||||
| `010 inbox/` | `note` | Unsorted placeholder |
|
||||
|
||||
Everything else has been **retired**: `subject`, `domain`, `tool`, `service`,
|
||||
`container`, `machine`, `storage`, `backup-system`, `network-component`,
|
||||
`website`, `person`. Do not reintroduce them.
|
||||
|
||||
### The lifecycle
|
||||
|
||||
```
|
||||
being built -> 200 projects/... -> project
|
||||
built and running-> 300 areas/... -> area
|
||||
```
|
||||
|
||||
A note moves between the two when the work finishes. Its `areas:` field records
|
||||
which area it belongs to while it is still a project.
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
|
||||
| Phase | Scope | State |
|
||||
|-------|-------|-------|
|
||||
| **A** | Vault scanner + deterministic classifier | ✅ |
|
||||
| **B** | JEV (TypeSafe) semantic tier | ✅ |
|
||||
| **C** | Writer: frontmatter, move, TOC insert, approval gate | ✅ |
|
||||
| D | MCP tool wrapper | pending |
|
||||
| E | Pi skill + Gitea distribution | pending |
|
||||
| F | Validation command | pending |
|
||||
|
||||
---
|
||||
|
||||
## Install
|
||||
|
||||
Nothing to install — pure standard library, Python 3.10+.
|
||||
|
||||
Set the API key for the semantic tier:
|
||||
|
||||
```bash
|
||||
export TYPESAFE_API_KEY=... # required for --jev / benchmark
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The workflow (how a note gets filed)
|
||||
|
||||
The watcher runs as a **systemd user service on `.27`**, always on.
|
||||
|
||||
```
|
||||
you save a note into 010 inbox/ type: note
|
||||
|
|
||||
v watcher notices within ~10s
|
||||
it classifies the note and WRITES THE PROPOSAL INTO THE NOTE
|
||||
|
||||
type: note
|
||||
proposed_folder: 200 projects/250 Electronics
|
||||
proposed_toc: 100 Table of Contents/Projects.md → Ideas
|
||||
proposed_tags: [solar, 12v, bird, bath]
|
||||
confidence: 0.69
|
||||
approved: false <-- a checkbox in Obsidian
|
||||
|
|
||||
v you open the note and tick the box (or edit the proposal first)
|
||||
approved: true
|
||||
|
|
||||
v watcher notices within ~10s
|
||||
it files the note, strips the admin fields, updates the TOC
|
||||
|
|
||||
v
|
||||
you see the note appear in the right folder, listed in the index
|
||||
```
|
||||
|
||||
**Nothing is ever filed without your approval.** You can:
|
||||
- tick `approved` to accept as-is
|
||||
- edit `proposed_folder` (or `proposed_tags`) first, then tick it
|
||||
- leave it alone — it waits in the inbox indefinitely
|
||||
|
||||
Admin fields (`proposed_folder`, `proposed_toc`, `proposed_tags`, `confidence`,
|
||||
`approved`) are removed when the note is filed, so the finished note is clean.
|
||||
|
||||
A running log is written to `010 inbox/_sort log.md`.
|
||||
|
||||
### Managing the watcher
|
||||
|
||||
```bash
|
||||
systemctl --user status obsidian-sorter # is it running?
|
||||
systemctl --user stop obsidian-sorter # pause it
|
||||
systemctl --user start obsidian-sorter # resume
|
||||
systemctl --user disable obsidian-sorter # don't start on boot
|
||||
journalctl --user -u obsidian-sorter -f # live output
|
||||
```
|
||||
|
||||
Logs: `~/.local/state/obsidian-sorter.log`
|
||||
|
||||
---
|
||||
|
||||
## Manual use
|
||||
|
||||
`obsidian-sort` is installed on `.27` (in `~/bin`):
|
||||
|
||||
```bash
|
||||
obsidian-sort # show what would happen (dry run, no writes)
|
||||
obsidian-sort apply # file the notes, asks to confirm
|
||||
obsidian-sort apply --yes # file without asking
|
||||
obsidian-sort scan # vault summary
|
||||
```
|
||||
|
||||
Or call the module directly from anywhere the code is deployed:
|
||||
|
||||
```bash
|
||||
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" watch --once
|
||||
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" sort-inbox --jev --apply
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Usage reference
|
||||
|
||||
### Read-only
|
||||
|
||||
```bash
|
||||
# Vault shape: counts, types, what needs classification
|
||||
python3 -m sorter --vault ~/obsidian scan
|
||||
|
||||
# Classify one note (rules only, free)
|
||||
python3 -m sorter --vault ~/obsidian propose "010 inbox/My Note.md"
|
||||
|
||||
# Classify with the semantic tier
|
||||
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
|
||||
propose "010 inbox/My Note.md" --jev
|
||||
```
|
||||
|
||||
### Filing (writes)
|
||||
|
||||
Dry run by default — nothing is written without `--apply`.
|
||||
|
||||
```bash
|
||||
# Show the plan
|
||||
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
|
||||
sort "010 inbox/My Note.md" --jev
|
||||
|
||||
# Apply it
|
||||
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
|
||||
sort "010 inbox/My Note.md" --jev --apply
|
||||
|
||||
# Whole inbox, no prompt (for agents)
|
||||
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
|
||||
sort-inbox --jev --apply --yes
|
||||
```
|
||||
|
||||
The plan lists every change explicitly:
|
||||
|
||||
```
|
||||
Plan for: 010 inbox/50 Legendary Websites.md
|
||||
|
||||
→ 300 areas/390 Web Pages and Tools To Try/50 Legendary Websites.md
|
||||
title : 50 Legendary Websites
|
||||
type : area status: active
|
||||
toc : 100 Table of Contents/Areas.md → Web Pages & Tools To Try
|
||||
|
||||
changes:
|
||||
+ create 300 areas/390 Web Pages and Tools To Try/50 Legendary Websites.md
|
||||
~ update 100 Table of Contents/Areas.md (add [[50 Legendary Websites]] under 'Web Pages & Tools To Try')
|
||||
- remove 010 inbox/50 Legendary Websites.md (moved out of inbox)
|
||||
```
|
||||
|
||||
### Measuring the semantic tier
|
||||
|
||||
```bash
|
||||
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" benchmark --all --quiet
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Safety guarantees (all tested)
|
||||
|
||||
| Guarantee | Behaviour |
|
||||
|-----------|-----------|
|
||||
| Dry run by default | `--apply` required to write anything |
|
||||
| `Untitled*` never moved | Plan marked BLOCKED, exit code 2, file untouched |
|
||||
| Idempotent TOC | Re-running never duplicates an index entry |
|
||||
| Never overwrites | An existing destination blocks the plan |
|
||||
| Frontmatter merged | Existing `tags`, `aliases`, `id` survive; `created` preserved |
|
||||
| Body untouched | The author's words are never rewritten |
|
||||
| Atomic writes | temp file + `os.replace` — no partial files |
|
||||
| `.trash/` off limits | Files under .trash are never removed |
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
Runs against an **isolated copy** — the mirror is never modified.
|
||||
|
||||
```bash
|
||||
python3 tests/run_tests.py # full suite (uses JEV)
|
||||
python3 tests/run_tests.py --no-jev # mechanics only, no API cost
|
||||
python3 tests/run_tests.py --keep # keep the temp vault for inspection
|
||||
```
|
||||
|
||||
Covers: dry-run safety, all three buckets routing to the correct sub-folder,
|
||||
frontmatter merging, the `Untitled` guard, destination collisions, `.trash`
|
||||
protection, inbox clearing, idempotency, and duplicate index entries.
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
# Read-only mirror of the live vault for testing
|
||||
rsync -a --delete \
|
||||
--exclude='.git' --exclude='.trash' --exclude='.venv' \
|
||||
--exclude='graphify-out*' --exclude='.obsidian' \
|
||||
sam@192.168.20.27:~/obsidian/ ./test-vault/
|
||||
```
|
||||
|
||||
`test-vault/` is a throwaway mirror and is gitignored.
|
||||
|
||||
### Layout
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `sorter/frontmatter.py` | Minimal YAML reader/writer (no PyYAML) |
|
||||
| `sorter/models.py` | Types, `Note`, `Proposal`, folder→type mapping |
|
||||
| `sorter/vault.py` | Vault scanning, index introspection |
|
||||
| `sorter/classify.py` | Tier 1 rules |
|
||||
| `sorter/jev.py` | TypeSafe JEV client + cost tracking |
|
||||
| `sorter/tier2.py` | Tier 2 semantic classifier + hybrid |
|
||||
| `sorter/writer.py` | Plans, frontmatter merge, TOC insert, apply |
|
||||
| `sorter/cli.py` | Command line |
|
||||
Reference in New Issue
Block a user