Tree-sitter Integration

Summary

Add a tree-sitter layer that provides code intelligence without requiring an LSP server. This gives the extension useful symbol search, document symbols, and definition lookup out of the box for any supported language — no install step needed.

Motivation

Today every tool in the extension requires a running LSP server. If the user hasn't installed typescript-language-server or pyright, they get nothing. Kiro CLI solves this with a built-in tree-sitter layer that covers 18 languages with zero setup.

Tree-sitter parsing is fast (incremental), works on syntactically broken files, and can power symbol search, document outlines, and basic go-to-definition without any external process.

Goals

Zero-config code intelligence — symbol search, document symbols, and definition lookup work immediately for supported languages
Graceful upgrade — when an LSP server is running, LSP results take priority; tree-sitter is the fallback
Codebase overview — expose a tool that summarizes project structure (directories, key files, entry points) using tree-sitter symbol extraction
Minimal footprint — tree-sitter WASM bindings, no native compilation required at install time

Non-goals

Replacing LSP — tree-sitter cannot do cross-file type resolution, find-references across files, or rename refactoring
Supporting every language immediately — start with the languages we already support via LSP (TypeScript, JavaScript, Python, Rust, Go, Java) and expand later

Design

Parser management

Use web-tree-sitter (WASM-based) so the extension works without native build tools
Bundle or lazy-download .wasm grammar files for each supported language
Cache parsed trees per-file; invalidate on didChange events from FileSync
Reuse the EXT_TO_LANGUAGE mapping in lsp-manager.ts to detect language from file extension

Fallback strategy

Each tool that currently calls the LSP checks whether a server is running for the file's language:

If LSP server is running and healthy → use LSP (current behavior)
If no LSP server → fall back to tree-sitter
If tree-sitter has no grammar for the language → return "no intelligence available" error

This logic lives in a shared resolveProvider(filePath) helper that tools call instead of going straight to manager.getClientForFile().

New tools / tool enhancements

Tool	Tree-sitter behavior
`lsp_symbols` (file)	Walk the tree-sitter AST for function, class, method, interface, enum, and variable declarations. Same output format as today.
`lsp_symbols` (workspace)	Scan project files (respecting `.gitignore`), parse each, and collect top-level symbols. Fuzzy-match against the query. Cache the index and update incrementally.
`lsp_definition`	For a symbol at a position, search the current file's tree for a matching definition node. If not found, search the workspace index. Best-effort — won't resolve imports through `node_modules` or type aliases.
`lsp_hover`	Extract the enclosing node's kind and text (e.g., "function declaration", the signature line). No type info — that requires LSP.
`lsp_diagnostics`	Tree-sitter can detect parse errors (syntax errors). Report those as a minimal diagnostic set when no LSP is available.
`code_overview`	New tool. Summarize project structure: directory tree, top-level symbols per file, entry points, dependency manifests. Uses tree-sitter for symbol extraction.

Workspace indexing

On first use, walk the project tree (skip node_modules, .git, build, dist, target, etc.)
Parse each file with tree-sitter, extract top-level symbols (name, kind, file, line)
Store in an in-memory index keyed by symbol name
Re-index individual files when FileSync reports changes
For large repos, cap initial indexing at ~5000 files and index remaining on demand

File layout

src/
├── tree-sitter/
│   ├── parser-manager.ts    # Load/cache WASM parsers per language
│   ├── symbol-extractor.ts  # AST → symbol list, per-language queries
│   ├── workspace-index.ts   # Project-wide symbol index
│   └── grammars/            # .wasm files (or download script)
├── resolve-provider.ts      # LSP-or-tree-sitter routing
└── tools/
    ├── code-overview.ts     # New tool
    └── ... (existing tools updated to use resolveProvider)

Open questions

Grammar distribution — bundle .wasm files in the repo (adds ~2-5 MB) or download on first use? Bundling is simpler; downloading keeps the repo small.
Incremental indexing performance — is the in-memory index fast enough for monorepos with 10k+ files, or do we need SQLite/a persistent cache?
Query language — tree-sitter has a built-in query language for pattern matching. Should we expose that as a user-facing feature here, or save it for the structural search spec?

4.9 KiB Raw Blame History