46 lines
3.1 KiB
Markdown
46 lines
3.1 KiB
Markdown
# dsh plugins — plan (docs / summarize / web / images)
|
||
|
||
Goal: give each family dsh instance practical "plugin-style" capabilities:
|
||
handle documents, summarize web pages & docs, ingest images, output images,
|
||
and edit images from text. Decide how to deliver them on our lightweight
|
||
FastAPI+SSE dsh.
|
||
|
||
## Approach: extend our dsh (not adopt the official harness)
|
||
Our dsh is intentionally a small FastAPI+SSE chat (OmniRoute `auto/best-chat`).
|
||
The official DeepSeek Harness + its ~11k-plugin ecosystem is heavier and
|
||
prerelease. We keep our app and add plugin-style capabilities natively, phased.
|
||
|
||
Capabilities map:
|
||
| # | Capability | Mechanism | Phase |
|
||
|---|------------|-----------|-------|
|
||
| 1 | Summarize a web page | dsh fetches URL (httpx) → text → LLM summary. Add `/api/tool/web` or a "summarize URL" box. | 1 |
|
||
| 2 | Summarize a document | read a file from `/workspace` (txt/md; PDF via pypdf) → LLM summary. Add "Summarize file" button + `/api/tool/docs`. | 1 |
|
||
| 3 | Handle / attach docs | pass referenced workspace file contents into the prompt (file picker, cap size). | 1–2 |
|
||
| 4 | Ingest images (understand) | need a **vision** model on OmniRoute (e.g. qwen-vl / gpt-4o). Send image as data-URI in OpenAI vision content format. `DSH_LLM_VISION_MODEL`. | 2 |
|
||
| 5 | Output images (generate) | OmniRoute is LLM-only → wire an image API (OpenRouter image, or internal ComfyUI/SD). `/api/tool/image`. | 3 |
|
||
| 6 | Edit image from text | image-editing model (instruction-based) on the image API. | 3 |
|
||
|
||
## Phase 1 — text capabilities (no new infra)
|
||
- [ ] Web summarize: `POST /api/tool/web {url}` → httpx fetch → strip HTML → LLM summary (stream). UI: a "Paste URL to summarize" box.
|
||
- [ ] Docs summarize: `POST /api/tool/docs {path}` (relative to /workspace) → read txt/md (pypdf for PDFs) → LLM summary.
|
||
- [ ] Chat context: pick a workspace file → prepend its contents to the message (cap ~8k tokens).
|
||
|
||
## Phase 2 — image ingest (vision)
|
||
- [ ] Find/configure a vision model on OmniRoute (qwen2.5-vl, gpt-4o, or similar). Set `DSH_LLM_VISION_MODEL`.
|
||
- [ ] Image upload in chat.html (`<input type=file accept=image/*>` + base64) → send OpenAI vision `content` array with `image_url` data-URI.
|
||
- [ ] Handle docs as images (scan/photo) → describe/OCR via the vision model.
|
||
|
||
## Phase 3 — image output & edit
|
||
- [ ] Wire an image-generation API (OpenRouter `gpt-image-1`/`flux`, or internal ComfyUI/SD webui).
|
||
- [ ] `/api/tool/image` — text→image; stream/provide a URL or return a base64 image to display in chat.
|
||
- [ ] Image edit — instruction/ref edit endpoint (gpt-image edit or SD img2img) from an uploaded image + text.
|
||
- [ ] Save images into the user's S3 bucket (`image/…`) so they're kept.
|
||
|
||
## Open questions
|
||
- Vision + image models on OmniRoute: confirm availability/ids (check `/v1/models`).
|
||
- Kids' instances: should image output be gated (cost/appropriateness)? Probably gate image-gen to sam initially.
|
||
- Marker: keep everything streaming + iframe-friendly CSP.
|
||
|
||
## Sources
|
||
- docs/dsh-plugins.md — shortlist from github topic: WeKnora (docs→RAG), modlens (vision), open-design.
|