obsidian-sorter/README.md

# obsidian-sorter

Automated note classification and filing for an Obsidian vault.

Notes enter through `010 inbox/`, the sorter decides where they belong, and —
once approved — writes them into place and updates the Table of Contents.

**Design goal: no dependencies.** The vault uses simple, uniform YAML; the tool
ships a small purpose-built parser so it deploys onto hosts without pip
(NixOS `.13`, `.27`, `.51`).

---

## 🛠 Demonstrated Skills & Tech Stack

- **Core Technologies**: Python 3.10+ using the **standard library only** — no
  PyYAML, no pip dependencies; systemd user services; YAML frontmatter.
- **Key Tools**: TypeSafe JEV (`POST api.typesafe.ai/v1/systemone`) for semantic
  classification, stdlib `urllib` for HTTP, `mmdc` + Archify for diagrams,
  `rsync` for deployment, Git + Gitea for version control.
- **Engineering Capabilities**: a deterministic rule engine with a probabilistic
  second tier, atomic file writes (`tempfile` + `os.replace`), idempotent index
  updates, a human-in-the-loop approval gate, and a purpose-built YAML parser
  written to avoid runtime dependencies on NixOS hosts.

## 📐 System Architecture

```mermaid
flowchart LR
    A["Sam writes a note"] --> B["010 inbox/"]
    B --> C{"Watcher<br/>systemd user service<br/>polls every 10s"}
    C --> D["Tier 1 — rules<br/>folder + keyword hints<br/>user tags win"]
    D -->|"confident"| F["Proposal"]
    D -->|"unsure"| E["Tier 2 — TypeSafe JEV<br/>bucket + folder + tags"]
    E --> F
    F --> G["Proposal written<br/>into the note frontmatter"]
    G --> H["Sam reviews in Obsidian"]
    H -->|"ticks approved"| I["Writer"]
    H -->|"edits folder first"| G
    I --> J["Destination folder"]
    I --> K["Table of Contents"]
```

*Interactive Architecture Map:* [maps.lab.audasmedia.com.au/obsidian-sorter/docs/obsidian-sorter-map.html](https://maps.lab.audasmedia.com.au/obsidian-sorter/docs/obsidian-sorter-map.html)

---

## The taxonomy (simplified 2026-09-24)

**The type of a note mirrors the folder it lives in.** Nothing is guessed from
content — the destination decides the type.

| Folder | Type | Meaning |
|--------|------|---------|
| `200 projects/` | `project` | Work in progress — something being built |
| `300 areas/` | `area` | Ongoing interest with documentation and notes |
| `400 Personal Family/` | `family` | Family / non-work |
| `100 Table of Contents/` | `index` | Navigation notes only |
| `000 daily/` | `daily` | Chronological daily notes |
| `500 archive/` | `archive` | Inactive material |
| `Clippings/` | `clipping` | Browser-extension captures |
| `010 inbox/` | `note` | Unsorted placeholder |

Everything else has been **retired**: `subject`, `domain`, `tool`, `service`,
`container`, `machine`, `storage`, `backup-system`, `network-component`,
`website`, `person`. Do not reintroduce them.

### The lifecycle

```
being built      ->  200 projects/...        ->  project
built and running->  300 areas/...           ->  area
```

A note moves between the two when the work finishes. Its `areas:` field records
which area it belongs to while it is still a project.

---

## Status

| Phase | Scope | State |
|-------|-------|-------|
| **A** | Vault scanner + deterministic classifier | ✅ |
| **B** | JEV (TypeSafe) semantic tier | ✅ |
| **C** | Writer: frontmatter, move, TOC insert, approval gate | ✅ |
| D | MCP tool wrapper | pending |
| E | Pi skill + Gitea distribution | pending |
| F | Validation command | pending |

---

## 🚀 Quick Start

Nothing to install — pure standard library, Python 3.10+.

Set the API key for the semantic tier:

```bash
export TYPESAFE_API_KEY=...        # required for --jev / benchmark
```

---

## The workflow (how a note gets filed)

The watcher runs as a **systemd user service on `.27`**, always on.

```
you save a note into 010 inbox/          type: note
        |
        v   watcher notices within ~10s
    it classifies the note and WRITES THE PROPOSAL INTO THE NOTE

        type: note
        proposed_folder: 200 projects/250 Electronics
        proposed_toc: 100 Table of Contents/Projects.md → Ideas
        proposed_tags: [solar, 12v, bird, bath]
        confidence: 0.69
        approved: false          <-- a checkbox in Obsidian
        |
        v   you open the note and tick the box (or edit the proposal first)
        approved: true
        |
        v   watcher notices within ~10s
    it files the note, strips the admin fields, updates the TOC
        |
        v
you see the note appear in the right folder, listed in the index
```

**Nothing is ever filed without your approval.** You can:
- tick `approved` to accept as-is
- edit `proposed_folder` (or `proposed_tags`) first, then tick it
- leave it alone — it waits in the inbox indefinitely

Admin fields (`proposed_folder`, `proposed_toc`, `proposed_tags`, `confidence`,
`approved`) are removed when the note is filed, so the finished note is clean.

A running log is written to `010 inbox/_sort log.md`.

### Managing the watcher

```bash
systemctl --user status  obsidian-sorter    # is it running?
systemctl --user stop    obsidian-sorter    # pause it
systemctl --user start   obsidian-sorter    # resume
systemctl --user disable obsidian-sorter    # don't start on boot
journalctl --user -u obsidian-sorter -f     # live output
```

Logs: `~/.local/state/obsidian-sorter.log`

---

## 📝 Usage & Configuration

### Manual use

`obsidian-sort` is installed on `.27` (in `~/bin`):

```bash
obsidian-sort              # show what would happen (dry run, no writes)
obsidian-sort apply        # file the notes, asks to confirm
obsidian-sort apply --yes  # file without asking
obsidian-sort scan         # vault summary
```

Or call the module directly from anywhere the code is deployed:

```bash
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" watch --once
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" sort-inbox --jev --apply
```

---

### Usage reference

#### Read-only

```bash
# Vault shape: counts, types, what needs classification
python3 -m sorter --vault ~/obsidian scan

# Classify one note (rules only, free)
python3 -m sorter --vault ~/obsidian propose "010 inbox/My Note.md"

# Classify with the semantic tier
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
    propose "010 inbox/My Note.md" --jev
```

#### Filing (writes)

Dry run by default — nothing is written without `--apply`.

```bash
# Show the plan
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
    sort "010 inbox/My Note.md" --jev

# Apply it
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
    sort "010 inbox/My Note.md" --jev --apply

# Whole inbox, no prompt (for agents)
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" \
    sort-inbox --jev --apply --yes
```

The plan lists every change explicitly:

```
Plan for: 010 inbox/50 Legendary Websites.md

  → 300 areas/390 Web Pages and Tools To Try/50 Legendary Websites.md
    title  : 50 Legendary Websites
    type   : area   status: active
    toc    : 100 Table of Contents/Areas.md → Web Pages & Tools To Try

  changes:
  + create  300 areas/390 Web Pages and Tools To Try/50 Legendary Websites.md
  ~ update  100 Table of Contents/Areas.md   (add [[50 Legendary Websites]] under 'Web Pages & Tools To Try')
  - remove  010 inbox/50 Legendary Websites.md   (moved out of inbox)
```

#### Measuring the semantic tier

```bash
python3 -m sorter --vault ~/obsidian --inbox "010 inbox" benchmark --all --quiet
```

---

## Safety guarantees (all tested)

| Guarantee | Behaviour |
|-----------|-----------|
| Dry run by default | `--apply` required to write anything |
| `Untitled*` never moved | Plan marked BLOCKED, exit code 2, file untouched |
| Idempotent TOC | Re-running never duplicates an index entry |
| Never overwrites | An existing destination blocks the plan |
| Frontmatter merged | Existing `tags`, `aliases`, `id` survive; `created` preserved |
| Body untouched | The author's words are never rewritten |
| Atomic writes | temp file + `os.replace` — no partial files |
| `.trash/` off limits | Files under .trash are never removed |

---

## Testing

Runs against an **isolated copy** — the mirror is never modified.

```bash
python3 tests/run_tests.py            # full suite (uses JEV)
python3 tests/run_tests.py --no-jev   # mechanics only, no API cost
python3 tests/run_tests.py --keep     # keep the temp vault for inspection
```

Covers: dry-run safety, all three buckets routing to the correct sub-folder,
frontmatter merging, the `Untitled` guard, destination collisions, `.trash`
protection, inbox clearing, idempotency, and duplicate index entries.

---

## Development

```bash
# Read-only mirror of the live vault for testing
rsync -a --delete \
  --exclude='.git' --exclude='.trash' --exclude='.venv' \
  --exclude='graphify-out*' --exclude='.obsidian' \
  sam@192.168.20.27:~/obsidian/ ./test-vault/
```

`test-vault/` is a throwaway mirror and is gitignored.

### Layout

| File | Purpose |
|------|---------|
| `sorter/frontmatter.py` | Minimal YAML reader/writer (no PyYAML) |
| `sorter/models.py` | Types, `Note`, `Proposal`, folder→type mapping |
| `sorter/vault.py` | Vault scanning, index introspection |
| `sorter/classify.py` | Tier 1 rules |
| `sorter/jev.py` | TypeSafe JEV client + cost tracking |
| `sorter/tier2.py` | Tier 2 semantic classifier + hybrid |
| `sorter/writer.py` | Plans, frontmatter merge, TOC insert, apply |
| `sorter/cli.py` | Command line |