# Phase 4 — Ingestion from sources

**Goal:** a fireable tool that reads the estate — Gitea, the notes vault, the knowledge graph — and keeps a searchable pool current. The pool is not built from `content/` alone; `content/` is the top tier of it.

**Depends on:** Phase 1 (content and schema), Phase 2 (FTS5 index).

**Why this changed.** An earlier version of this document said the corpus would be built from `content/` only. That was wrong. A hand-curated set cannot answer "how did the photo pipeline deduplicate" or "what did the garden controller do before the rewrite". The estate can. Curated claims answer questions about Sam; the estate answers questions about the work.

---

## 1. Sources

| Source | Reached by | What to take |
|---|---|---|
| **Gitea**, all repositories | API over HTTPS | `README.md`, file tree (paths only), selected source files, recent commit messages, language statistics, repo description and topics |
| **`sam/obsidian-vault`** | Same Gitea API — the vault is a repo | Markdown body, front-matter title and tags, folder path |
| **Graphify output** | `graph.json` committed to the vault repo | Nodes (label, source_file, community), edges (relation, confidence) |
| **`content/`** | Local repo | The curated claims. **Tier 1** — outranks everything above on conflict |

Gitea is the only network dependency. The vault and the graph arrive through it, so there is no cross-machine copy and no SSH from `.13` to `.27`.

### What not to read

Media never becomes text. Images, video and audio are recorded as `image` node references, tagged with their path, and nothing more.

---

## 2. Platform guides

One file per platform in `platforms/<name>.yml`. They are configuration **and** public copy — the site shows them so a visitor can see what the assistant reads.

```yaml
name: gitea
enabled: true
what: "Self-hosted Git. Source for every current project."
fits: [ai, platform, edge]
base: https://gitea.lab.audasmedia.com.au
auth: GITEA_TOKEN
include:
  repos: all
exclude:
  paths: ["**/node_modules/**", "**/.venv/**", "**/.git/**", "**/*.env*",
          "**/.silverbullet.auth.json", "**/*.pem", "**/*.key"]
  repos: []
read: [readme, tree, selected_source, commits, languages]
limits: { max_documents: 4000, max_file_kb: 200 }
```

```yaml
name: obsidian
enabled: true
what: "Personal knowledge base. High-level notes on every system, in Sam's words."
fits: [about, ai, platform, edge]
base: sam/obsidian-vault
include:
  paths: ["200 projects/**", "300 areas/**", "400 Personal Family/**",
          "100 Table of Contents/**", "010 inbox/**"]
exclude:
  paths: [".obsidian/**", ".trash/**", ".venv/**", "templates/**",
          "**/*.excalidraw", "**/*.png", "**/*.jpg"]
limits: { max_documents: 4000, max_file_kb: 200 }
```

```yaml
name: graphify
enabled: true
what: "Knowledge graph over the vault. Answers 'what connects X to Y' rather than 'what mentions X'."
fits: [ai]
base: sam/obsidian-vault
read: [graph_json]
```

`fits` drives the payload `area`, which is what lets the query filter before it searches.

---

## 3. Metadata on every chunk

```json
{
  "platform": "gitea | obsidian | graphify | curated",
  "tier": 1,
  "source_type": "doc | code | commit | graph_node | claim",
  "area": "ai | platform | edge | products | history | about",
  "origin_url": "https://gitea.../sam/photo-pipeline/src/x.py",
  "path": "src/x.py",
  "title": "photo-pipeline / dedup",
  "confidence": "delivered | running | in-progress | abandoned | historic",
  "last_modified": "2026-10-14T05:00:00Z",
  "content_hash": "sha256:...",
  "public": true,
  "text": "..."
}
```

`tier: 1` is curated `content/`. Everything harvested is `tier: 2`. The query layer uses this to prefer the curated claim when the two disagree.

`origin_url` is mandatory for harvested chunks. A chunk with no origin is not admitted — an answer that cannot be traced is worse than no answer.

---

## 4. Pipeline

```
guides  →  harvest (per platform)  →  normalise to text  →  chunk  →  enrich  →  embed  →  upsert
                                        │
                                        └─ reject: excluded path, no origin, binary, oversize
```

- **Chunking** is code-aware: 400–800 tokens, ~12% overlap, never split inside a fenced block, table, front-matter block or function. Use the language grammar where available; fall back to paragraph splitting.
- **Deterministic IDs:** `sha256(origin_url + chunk_index + content_hash)`. Re-running updates, never duplicates.
- **Incremental:** compare `content_hash` against what the pool already holds; embed only what changed. A full rebuild must always be possible and is one command.
- **Graph** is not chunked. Nodes and edges go into their own collection, joined to note chunks by `path`, so a graph answer can name the note it came from.

---

## 5. Scheduling

- On demand: one command.
- Scheduled: a Prefect deployment on the existing server. Worker only; do not deploy a second Prefect server.
- Reports what changed each run: documents added, updated, removed, and any excluded-path rejections.
- A source that has not changed in its expected window raises an alert through the existing Apprise service. Silent staleness is the failure mode that matters.

---

## 6. Definition of Done

- [ ] One command rebuilds the pool from empty.
- [ ] A second run produces no duplicate vectors.
- [ ] A probe document placed in an excluded path never appears in the pool, and the run reports the rejection.
- [ ] A document with no resolvable origin is refused, not defaulted.
- [ ] Every chunk carries all fields in section 3.
- [ ] Incremental mode re-embeds only changed documents, proven by embedding call count.
- [ ] Graph nodes and edges are queryable and each resolves to a source note.
- [ ] The scheduled run reports deltas and alerts on staleness.
- [ ] Qdrant stays inside its `mem_limit`; footprint recorded at 1k, 10k and 40k chunks.
- [ ] No credential, `.env`, key file or family-personal binary is present in the pool — verified by search, not by assertion.

---

## 7. Out of scope

- Answering questions. This phase ends at "retrieval returns the right chunks with their origins".
- The query graph, prompts and guardrails — Phase 5.
- Building the Graphify graph. It already exists and is rebuilt on `.27` with its own skill. This phase consumes its output.

---

## 8. Risks

| Risk | Mitigation |
|---|---|
| Volume — 50 repos plus 280 notes exceeds the memory cap | Cap per platform, skip large files, measure at three corpus sizes |
| Scraped text contradicts a curated claim | `tier` field; the query prefers tier 1 and says so |
| A secret is harvested | Exclusion list plus a post-run scan of the pool for credential patterns, failing the run if found |
| Staleness is invisible | Scheduled run plus alert |
| The estate is unavailable when a recruiter asks | The pool is local; only ingestion needs Gitea. Serving never does |