3.1 KiB
3.1 KiB
dsh plugins — plan (docs / summarize / web / images)
Goal: give each family dsh instance practical "plugin-style" capabilities: handle documents, summarize web pages & docs, ingest images, output images, and edit images from text. Decide how to deliver them on our lightweight FastAPI+SSE dsh.
Approach: extend our dsh (not adopt the official harness)
Our dsh is intentionally a small FastAPI+SSE chat (OmniRoute auto/best-chat).
The official DeepSeek Harness + its ~11k-plugin ecosystem is heavier and
prerelease. We keep our app and add plugin-style capabilities natively, phased.
Capabilities map:
| # | Capability | Mechanism | Phase |
|---|---|---|---|
| 1 | Summarize a web page | dsh fetches URL (httpx) → text → LLM summary. Add /api/tool/web or a "summarize URL" box. |
1 |
| 2 | Summarize a document | read a file from /workspace (txt/md; PDF via pypdf) → LLM summary. Add "Summarize file" button + /api/tool/docs. |
1 |
| 3 | Handle / attach docs | pass referenced workspace file contents into the prompt (file picker, cap size). | 1–2 |
| 4 | Ingest images (understand) | need a vision model on OmniRoute (e.g. qwen-vl / gpt-4o). Send image as data-URI in OpenAI vision content format. DSH_LLM_VISION_MODEL. |
2 |
| 5 | Output images (generate) | OmniRoute is LLM-only → wire an image API (OpenRouter image, or internal ComfyUI/SD). /api/tool/image. |
3 |
| 6 | Edit image from text | image-editing model (instruction-based) on the image API. | 3 |
Phase 1 — text capabilities (no new infra)
- Web summarize:
POST /api/tool/web {url}→ httpx fetch → strip HTML → LLM summary (stream). UI: a "Paste URL to summarize" box. - Docs summarize:
POST /api/tool/docs {path}(relative to /workspace) → read txt/md (pypdf for PDFs) → LLM summary. - Chat context: pick a workspace file → prepend its contents to the message (cap ~8k tokens).
Phase 2 — image ingest (vision)
- Find/configure a vision model on OmniRoute (qwen2.5-vl, gpt-4o, or similar). Set
DSH_LLM_VISION_MODEL. - Image upload in chat.html (
<input type=file accept=image/*>+ base64) → send OpenAI visioncontentarray withimage_urldata-URI. - Handle docs as images (scan/photo) → describe/OCR via the vision model.
Phase 3 — image output & edit
- Wire an image-generation API (OpenRouter
gpt-image-1/flux, or internal ComfyUI/SD webui). /api/tool/image— text→image; stream/provide a URL or return a base64 image to display in chat.- Image edit — instruction/ref edit endpoint (gpt-image edit or SD img2img) from an uploaded image + text.
- Save images into the user's S3 bucket (
image/…) so they're kept.
Open questions
- Vision + image models on OmniRoute: confirm availability/ids (check
/v1/models). - Kids' instances: should image output be gated (cost/appropriateness)? Probably gate image-gen to sam initially.
- Marker: keep everything streaming + iframe-friendly CSP.
Sources
- docs/dsh-plugins.md — shortlist from github topic: WeKnora (docs→RAG), modlens (vision), open-design.