Add pi-langfuse extension (Langfuse LLM observability for pi) — vendored from npm 1.5.8
This commit is contained in:
278
extensions/pi-langfuse/README.md
Normal file
278
extensions/pi-langfuse/README.md
Normal file
@@ -0,0 +1,278 @@
|
||||
# pi-langfuse
|
||||
|
||||
[](https://www.npmjs.com/package/pi-langfuse)
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
|
||||
[**English**](./README.md) | [**简体中文**](./README_CN.md)
|
||||
|
||||
Langfuse observability extension for [Pi Coding Agent](https://github.com/earendil-works/pi-coding-agent). It sends complete Pi runs to [Langfuse](https://langfuse.com) so the prompt, agent workflow, LLM generations, tool calls, final response, usage, cost, and health scores appear in one trace.
|
||||
|
||||
## What This Adds to Pi
|
||||
|
||||
- One Langfuse trace per user prompt, grouped by Pi session.
|
||||
- Root `agent`, per-request `generation`, and per-tool `tool` observations.
|
||||
- Final assistant output capture, tool error visibility, and trace-level scores.
|
||||
- Privacy controls for inputs, outputs, tool I/O, system prompt, and cwd.
|
||||
- Secret redaction and local path hashing before upload.
|
||||
- REST fallback for self-hosted Langfuse setups where OTel spans arrive but traces do not materialize.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Node.js** >= 22
|
||||
- **Pi Coding Agent** installed and configured
|
||||
- A **Langfuse** account ([cloud](https://cloud.langfuse.com) or self-hosted)
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. Install the extension:
|
||||
|
||||
```bash
|
||||
pi install npm:pi-langfuse
|
||||
```
|
||||
|
||||
2. Run Pi once. If no credentials are configured yet, Pi prompts for:
|
||||
- Langfuse public key, starting with `pk-lf-...`
|
||||
- Langfuse secret key, starting with `sk-lf-...`
|
||||
- Langfuse host, defaulting to `https://cloud.langfuse.com`
|
||||
|
||||
3. Run Pi normally:
|
||||
|
||||
```bash
|
||||
pi "Explain the architecture of Redis"
|
||||
```
|
||||
|
||||
4. Open Langfuse and inspect the new trace.
|
||||
|
||||
## Configuration
|
||||
|
||||
Langfuse API keys are available in **Langfuse Cloud** -> **Settings** -> **API Keys**.
|
||||
|
||||
### Method 1: Interactive setup
|
||||
|
||||
Run any `pi` command with the extension loaded. On first run without configuration, Pi prompts in the CLI or TUI and saves the result to `~/.pi/agent/pi-langfuse/config.json`.
|
||||
|
||||
To run setup again:
|
||||
|
||||
```text
|
||||
/langfuse-setup
|
||||
```
|
||||
|
||||
To inspect the active configuration without exposing secrets:
|
||||
|
||||
```text
|
||||
/langfuse-status
|
||||
```
|
||||
|
||||
The status command reports the config source, host, masked public key, capture policy, active-run state, config path, and last runtime error.
|
||||
|
||||
### Method 2: Environment variables
|
||||
|
||||
Set these before starting Pi:
|
||||
|
||||
```bash
|
||||
export LANGFUSE_PUBLIC_KEY="pk-lf-xxxx"
|
||||
export LANGFUSE_SECRET_KEY="sk-lf-xxxx"
|
||||
export LANGFUSE_BASE_URL="https://cloud.langfuse.com" # optional; LANGFUSE_HOST is also supported
|
||||
```
|
||||
|
||||
Saved config takes precedence. Environment variables are only used when `~/.pi/agent/pi-langfuse/config.json` is missing or incomplete.
|
||||
|
||||
Privacy controls can also be set through environment variables:
|
||||
|
||||
```bash
|
||||
export LANGFUSE_PRIVACY_PRESET="full-debug"
|
||||
```
|
||||
|
||||
Available presets:
|
||||
|
||||
| Preset | Captures |
|
||||
|--------|----------|
|
||||
| `metadata-only` | Metadata only; omits inputs, outputs, tool I/O, system prompt, and cwd |
|
||||
| `prompts-only` | Prompt/provider inputs plus metadata |
|
||||
| `conversations` | Inputs and assistant outputs, but omits tool I/O, system prompt, and cwd |
|
||||
| `full-debug` | Full trace detail; this is the default |
|
||||
|
||||
Fine-grained flags override presets:
|
||||
|
||||
```bash
|
||||
export LANGFUSE_CAPTURE_INPUTS=true
|
||||
export LANGFUSE_CAPTURE_OUTPUTS=true
|
||||
export LANGFUSE_CAPTURE_TOOL_IO=false
|
||||
export LANGFUSE_CAPTURE_SYSTEM_PROMPT=false
|
||||
export LANGFUSE_CAPTURE_CWD=false
|
||||
```
|
||||
|
||||
All captured payloads are redacted before upload. The extension masks common API keys, bearer tokens, passwords, cookies, private keys, Langfuse keys, GitHub/npm/AWS-style tokens, and local absolute paths.
|
||||
|
||||
### Payload limits
|
||||
|
||||
Before upload, payloads are shaped: strings are truncated and deeply nested or
|
||||
very wide structures are trimmed. These caps keep traces small and protect the
|
||||
Langfuse ingestion pipeline. Override any of them (no rebuild needed):
|
||||
|
||||
```bash
|
||||
export PI_LANGFUSE_MAX_STRING_LENGTH=12000 # per-string chars (system prompt, inputs)
|
||||
export PI_LANGFUSE_MAX_TOOL_PAYLOAD_LENGTH=24000 # per tool input/output chars
|
||||
export PI_LANGFUSE_MAX_DEPTH=6 # max nesting depth
|
||||
export PI_LANGFUSE_MAX_ARRAY_ITEMS=50 # max array elements kept
|
||||
export PI_LANGFUSE_MAX_OBJECT_KEYS=80 # max object keys kept
|
||||
export PI_LANGFUSE_MAX_PAYLOAD_NODES=2000 # max total nodes per payload
|
||||
```
|
||||
|
||||
Set any limit to `0`, `off`, `none`, or `unlimited` to disable that cap
|
||||
entirely (captures the full value). Unset or invalid values fall back to the
|
||||
defaults shown above. To capture a very large system prompt or big tool
|
||||
payloads in full, raise or disable the relevant limit (e.g.
|
||||
`PI_LANGFUSE_MAX_STRING_LENGTH=off`).
|
||||
|
||||
### Method 3: Persistent `config.json`
|
||||
|
||||
Create or update `~/.pi/agent/pi-langfuse/config.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"publicKey": "pk-lf-xxxx",
|
||||
"secretKey": "sk-lf-xxxx",
|
||||
"host": "https://cloud.langfuse.com",
|
||||
"privacyPreset": "conversations"
|
||||
}
|
||||
```
|
||||
|
||||
Fine-grained capture flags can also be persisted:
|
||||
|
||||
```json
|
||||
{
|
||||
"publicKey": "pk-lf-xxxx",
|
||||
"secretKey": "sk-lf-xxxx",
|
||||
"host": "https://cloud.langfuse.com",
|
||||
"capture": {
|
||||
"LANGFUSE_PRIVACY_PRESET": "metadata-only",
|
||||
"LANGFUSE_CAPTURE_INPUTS": "true"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Security**: Keep `~/.pi/agent/pi-langfuse/config.json` private. Never commit API keys to version control.
|
||||
> When the extension writes this file itself, it creates the config directory with `0700` permissions and the file with `0600` permissions where the host filesystem supports POSIX modes.
|
||||
|
||||
## Verify the Extension
|
||||
|
||||
Check that Pi has loaded the package:
|
||||
|
||||
```bash
|
||||
pi list
|
||||
```
|
||||
|
||||
`pi-langfuse` should appear in the installed package list.
|
||||
|
||||
To verify the Langfuse host and API keys from inside Pi, run:
|
||||
|
||||
```text
|
||||
/langfuse-test
|
||||
```
|
||||
|
||||
This command makes a timeout-bounded authenticated request to Langfuse and, if it succeeds, sends a small test trace.
|
||||
|
||||
## What Appears in Langfuse
|
||||
|
||||
- Each Pi session gets its own Langfuse session ID.
|
||||
- Each user prompt within that session becomes a separate trace.
|
||||
- The trace contains the final assistant output shown in Pi.
|
||||
- Tool runs appear as tool observations with arguments, results, and error state.
|
||||
- LLM requests appear as generation observations, including usage and cost when the provider exposes them.
|
||||
- Trace-level scores include tool counts, tool success rate, and whether the run had errors.
|
||||
|
||||
The package also includes a Langfuse CLI skill, so Langfuse data can be queried directly from Pi:
|
||||
|
||||
```text
|
||||
/pi-langfuse-langfuse <your-query>
|
||||
```
|
||||
|
||||
## Source Metadata
|
||||
|
||||
Local prototype note: source metadata support in this installed package is a local prototype patch. A durable solution should be shipped through an upstream PR, a fork, or a maintained package version so reinstalling the extension does not lose the behavior.
|
||||
|
||||
For Git-backed runs, the extension attaches safe source metadata to traces:
|
||||
|
||||
```json
|
||||
{
|
||||
"source_type": "git-repo",
|
||||
"repo_identity": "owner/repo",
|
||||
"repo_owner": "owner",
|
||||
"repo_name": "repo",
|
||||
"repo_root_name": "repo",
|
||||
"git_branch": "main",
|
||||
"git_commit": "abc123",
|
||||
"git_remote_host": "github.com",
|
||||
"git_remote_path": "owner/repo",
|
||||
"metadata_source": "git-detection"
|
||||
}
|
||||
```
|
||||
|
||||
`repo_identity` is `owner/repo`. `repo_name` is the repo name only and must not contain a slash.
|
||||
|
||||
A Git repo may optionally provide `.pi-langfuse.metadata.json`. Overrides are whitelist-only; unknown keys are ignored. Allowed keys are:
|
||||
|
||||
```text
|
||||
repo_identity
|
||||
repo_owner
|
||||
repo_name
|
||||
source_type
|
||||
service_name
|
||||
project_slug
|
||||
environment
|
||||
observability_owner
|
||||
```
|
||||
|
||||
Repo-local overrides are used only after the working directory is confirmed to be inside a usable Git repo. If Git detection fails for any reason, including a missing Git command, corrupted repo, or non-Git folder, the extension ignores repo-local identity files and emits only:
|
||||
|
||||
```json
|
||||
{
|
||||
"source_type": "non-git",
|
||||
"metadata_source": "non-git"
|
||||
}
|
||||
```
|
||||
|
||||
The extension must not upload raw absolute local paths, credentialed remotes, tokens, unknown override keys, or folder names for non-Git folders.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No traces appearing?
|
||||
|
||||
- Verify the API keys and run `/langfuse-setup` again if needed.
|
||||
- Run `/langfuse-status` to confirm the loaded host, config source, privacy mode, and last runtime error.
|
||||
- Confirm the Langfuse project is active and accepts writes.
|
||||
- Confirm the keys have write permission.
|
||||
- Look for `📊 Langfuse:` log messages in Pi output.
|
||||
|
||||
### Extension not loading?
|
||||
|
||||
```bash
|
||||
pi list
|
||||
pi install npm:pi-langfuse
|
||||
```
|
||||
|
||||
### "Missing config" on startup?
|
||||
|
||||
- Run `/langfuse-setup`.
|
||||
- Or set `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` before starting Pi.
|
||||
|
||||
### Model or cost not showing?
|
||||
|
||||
- Some providers do not expose cost information.
|
||||
- Inspect the raw observation data in Langfuse traces.
|
||||
- The `model` field can come from provider events, finalized assistant messages, `model_select`, or `ctx.model`.
|
||||
|
||||
### API key errors?
|
||||
|
||||
- Public keys start with `pk-lf-`.
|
||||
- Secret keys start with `sk-lf-`.
|
||||
- For self-hosted deployments, verify the host URL.
|
||||
|
||||
## Development Docs
|
||||
|
||||
Development setup, source installation, runtime architecture, trace model, tracked fields, and validation steps are documented in [DEVELOPMENT.md](./DEVELOPMENT.md) and [DEVELOPMENT_CN.md](./DEVELOPMENT_CN.md).
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
210
extensions/pi-langfuse/README_CN.md
Normal file
210
extensions/pi-langfuse/README_CN.md
Normal file
@@ -0,0 +1,210 @@
|
||||
# pi-langfuse
|
||||
|
||||
[](https://www.npmjs.com/package/pi-langfuse)
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
|
||||
[**English**](./README.md) | [**简体中文**](./README_CN.md)
|
||||
|
||||
[Pi Coding Agent](https://github.com/earendil-works/pi-coding-agent) 的 Langfuse 可观测性扩展。它会将完整的 Pi 运行发送到 [Langfuse](https://langfuse.com)，在一个 trace 中展示提示词、代理工作流、LLM 生成、工具调用、最终回复、用量、成本和健康分数。
|
||||
|
||||
## 这个插件提供什么
|
||||
|
||||
- 每个用户提示词对应一个 Langfuse trace，并按 Pi 会话分组。
|
||||
- 为根代理创建 `agent` 观察节点，为每次模型请求创建 `generation`，为每次工具调用创建 `tool`。
|
||||
- 记录最终助手输出、工具错误状态和追踪级别分数。
|
||||
- 提供输入、输出、工具 I/O、system prompt 和 cwd 的隐私采集开关。
|
||||
- 上传前脱敏常见密钥，并对本地绝对路径做 hash。
|
||||
- 针对自托管 Langfuse 提供 REST 兜底，覆盖 OTel span 已到达但 trace 未可见的场景。
|
||||
|
||||
## 前提条件
|
||||
|
||||
- **Node.js** >= 22
|
||||
- **Pi Coding Agent** 已安装并完成基础配置
|
||||
- **Langfuse** 账户，支持 [云服务](https://cloud.langfuse.com) 和自托管
|
||||
|
||||
## 快速开始
|
||||
|
||||
1. 安装扩展：
|
||||
|
||||
```bash
|
||||
pi install npm:pi-langfuse
|
||||
```
|
||||
|
||||
2. 首次运行 Pi 时，如果尚未配置凭据，Pi 会提示输入：
|
||||
- Langfuse 公钥，以 `pk-lf-...` 开头
|
||||
- Langfuse 密钥，以 `sk-lf-...` 开头
|
||||
- Langfuse 主机地址，默认 `https://cloud.langfuse.com`
|
||||
|
||||
3. 正常运行 Pi：
|
||||
|
||||
```bash
|
||||
pi "解释 Redis 的架构"
|
||||
```
|
||||
|
||||
4. 打开 Langfuse，查看新生成的 trace。
|
||||
|
||||
## 配置
|
||||
|
||||
Langfuse API 密钥可在 **Langfuse Cloud** -> **Settings** -> **API Keys** 中获取。
|
||||
|
||||
### 方式 1：交互式设置
|
||||
|
||||
加载扩展后运行任意 `pi` 命令。首次运行且未配置时，Pi 会在 CLI 或 TUI 中提示输入，并将结果保存到 `~/.pi/agent/pi-langfuse/config.json`。
|
||||
|
||||
如需重新执行设置：
|
||||
|
||||
```text
|
||||
/langfuse-setup
|
||||
```
|
||||
|
||||
如需查看当前配置状态且不泄漏密钥：
|
||||
|
||||
```text
|
||||
/langfuse-status
|
||||
```
|
||||
|
||||
状态命令会显示配置来源、主机地址、脱敏后的公钥、采集策略、是否有活跃运行、配置文件路径，以及最近一次运行时错误。
|
||||
|
||||
### 方式 2：环境变量
|
||||
|
||||
在启动 Pi 前设置：
|
||||
|
||||
```bash
|
||||
export LANGFUSE_PUBLIC_KEY="pk-lf-xxxx"
|
||||
export LANGFUSE_SECRET_KEY="sk-lf-xxxx"
|
||||
export LANGFUSE_BASE_URL="https://cloud.langfuse.com" # 可选；也支持 LANGFUSE_HOST
|
||||
```
|
||||
|
||||
保存的配置优先级更高。只有当 `~/.pi/agent/pi-langfuse/config.json` 缺失或不完整时，扩展才会使用环境变量。
|
||||
|
||||
隐私采集策略也可以通过环境变量设置：
|
||||
|
||||
```bash
|
||||
export LANGFUSE_PRIVACY_PRESET="full-debug"
|
||||
```
|
||||
|
||||
可用预设：
|
||||
|
||||
| 预设 | 采集内容 |
|
||||
|------|----------|
|
||||
| `metadata-only` | 仅采集元数据；不采集输入、输出、工具 I/O、system prompt 和 cwd |
|
||||
| `prompts-only` | 采集提示词或提供商输入，以及元数据 |
|
||||
| `conversations` | 采集输入和助手输出，但不采集工具 I/O、system prompt 和 cwd |
|
||||
| `full-debug` | 完整追踪细节；默认值 |
|
||||
|
||||
细粒度开关会覆盖预设：
|
||||
|
||||
```bash
|
||||
export LANGFUSE_CAPTURE_INPUTS=true
|
||||
export LANGFUSE_CAPTURE_OUTPUTS=true
|
||||
export LANGFUSE_CAPTURE_TOOL_IO=false
|
||||
export LANGFUSE_CAPTURE_SYSTEM_PROMPT=false
|
||||
export LANGFUSE_CAPTURE_CWD=false
|
||||
```
|
||||
|
||||
所有被采集的负载在上传前仍会脱敏。扩展会隐藏常见 API key、Bearer token、密码、Cookie、私钥、Langfuse key、GitHub/npm/AWS 风格 token，并对本地绝对路径做 hash。
|
||||
|
||||
### 方式 3：持久化 `config.json`
|
||||
|
||||
创建或更新 `~/.pi/agent/pi-langfuse/config.json`：
|
||||
|
||||
```json
|
||||
{
|
||||
"publicKey": "pk-lf-xxxx",
|
||||
"secretKey": "sk-lf-xxxx",
|
||||
"host": "https://cloud.langfuse.com",
|
||||
"privacyPreset": "conversations"
|
||||
}
|
||||
```
|
||||
|
||||
也可以持久化细粒度采集开关：
|
||||
|
||||
```json
|
||||
{
|
||||
"publicKey": "pk-lf-xxxx",
|
||||
"secretKey": "sk-lf-xxxx",
|
||||
"host": "https://cloud.langfuse.com",
|
||||
"capture": {
|
||||
"LANGFUSE_PRIVACY_PRESET": "metadata-only",
|
||||
"LANGFUSE_CAPTURE_INPUTS": "true"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **安全提醒**：`~/.pi/agent/pi-langfuse/config.json` 包含敏感信息，不应提交到版本控制。
|
||||
> 扩展自行写入该文件时，会在支持 POSIX 权限的文件系统上使用 `0700` 创建配置目录，并使用 `0600` 写入配置文件。
|
||||
|
||||
## 验证扩展是否已加载
|
||||
|
||||
执行：
|
||||
|
||||
```bash
|
||||
pi list
|
||||
```
|
||||
|
||||
已安装包列表中应出现 `pi-langfuse`。
|
||||
|
||||
如需在 Pi 内验证 Langfuse 主机地址和 API key：
|
||||
|
||||
```text
|
||||
/langfuse-test
|
||||
```
|
||||
|
||||
该命令会先发起一次带超时的认证请求；认证通过后，再发送一条小的测试 trace。
|
||||
|
||||
## 在 Langfuse 中会看到什么
|
||||
|
||||
- 每个 Pi 会话对应一个独立的 Langfuse session ID。
|
||||
- 该会话中的每个用户提示词都会生成一个独立 trace。
|
||||
- trace 中会包含 Pi 实际显示的最终助手回复。
|
||||
- 工具执行会以工具观察节点展示参数、结果和错误状态。
|
||||
- 模型请求会以生成观察节点展示；如果提供商暴露相关信息，还会包含用量和成本。
|
||||
- trace 级别会记录工具调用次数、工具成功率和是否出现错误。
|
||||
|
||||
此包还包含一个内置 Langfuse 技能，可直接在 Pi 中查询 Langfuse 数据：
|
||||
|
||||
```text
|
||||
/pi-langfuse-langfuse <查询内容>
|
||||
```
|
||||
|
||||
## 故障排除
|
||||
|
||||
### 没有看到 trace
|
||||
|
||||
- 先检查 API 密钥是否正确，必要时重新执行 `/langfuse-setup`。
|
||||
- 执行 `/langfuse-status`，确认当前加载的主机、配置来源、隐私模式和最近一次运行时错误。
|
||||
- 确认 Langfuse 项目处于可写状态。
|
||||
- 确认密钥具备写权限。
|
||||
- 在 Pi 输出中查找 `📊 Langfuse:` 日志。
|
||||
|
||||
### 扩展未加载
|
||||
|
||||
```bash
|
||||
pi list
|
||||
pi install npm:pi-langfuse
|
||||
```
|
||||
|
||||
### 启动时显示 `Missing config`
|
||||
|
||||
- 执行 `/langfuse-setup`。
|
||||
- 或在启动 Pi 前设置 `LANGFUSE_PUBLIC_KEY` 和 `LANGFUSE_SECRET_KEY`。
|
||||
|
||||
### 模型或成本未显示
|
||||
|
||||
- 并非所有提供商都会返回成本信息。
|
||||
- 可在 Langfuse trace 中查看原始观察数据。
|
||||
- `model` 字段可能来自提供商事件、已定型的助手消息、`model_select` 或 `ctx.model`。
|
||||
|
||||
### API 密钥错误
|
||||
|
||||
- 公钥以 `pk-lf-` 开头。
|
||||
- 密钥以 `sk-lf-` 开头。
|
||||
- 使用自托管时，还需要确认主机地址是否正确。
|
||||
|
||||
## 开发文档
|
||||
|
||||
源码安装、开发流程、运行时架构、追踪模型、字段明细和验证步骤已迁移到 [DEVELOPMENT.md](./DEVELOPMENT.md) 与 [DEVELOPMENT_CN.md](./DEVELOPMENT_CN.md)。
|
||||
|
||||
## 许可证
|
||||
|
||||
MIT
|
||||
BIN
extensions/pi-langfuse/image.png
Normal file
BIN
extensions/pi-langfuse/image.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 132 KiB |
215
extensions/pi-langfuse/index.ts
Normal file
215
extensions/pi-langfuse/index.ts
Normal file
@@ -0,0 +1,215 @@
|
||||
/**
|
||||
* Langfuse Observability Extension for Pi Coding Agent
|
||||
*
|
||||
* Sends one complete Langfuse trace per Pi agent run:
|
||||
* - root agent observation for the user prompt and final assistant response
|
||||
* - one generation observation per provider request
|
||||
* - one tool observation per tool call, keyed by toolCallId
|
||||
*/
|
||||
|
||||
import { basename } from "node:path";
|
||||
import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
|
||||
|
||||
import { state, resetRunState, runWithSession, setCurrentSession } from "./src/state.js";
|
||||
import { ensureConfig, promptForConfig, loadConfig } from "./src/config.js";
|
||||
import { shutdownRuntime } from "./src/langfuse.js";
|
||||
import { handleLangfusePrivacyCommand, handleLangfuseStatusCommand, handleLangfuseTestCommand } from "./src/commands.js";
|
||||
import { getMessageFromEvent, extractAssistantOutput, getCapturePolicy } from "./src/utils.js";
|
||||
import { applyCapturePolicy } from "./src/capture-policy.js";
|
||||
import { startAgentRun, finishAgentRun } from "./src/handlers/agent.js";
|
||||
import { startTurnObservation, finishTurnObservation } from "./src/handlers/turn.js";
|
||||
import {
|
||||
startGeneration,
|
||||
updateGenerationMetadata,
|
||||
finishGenerationFromMessage,
|
||||
createFallbackGenerationFromTurn,
|
||||
recordTTFT,
|
||||
} from "./src/handlers/generation.js";
|
||||
import {
|
||||
startToolObservation,
|
||||
finishToolObservation,
|
||||
closeDanglingObservations,
|
||||
} from "./src/handlers/tool.js";
|
||||
|
||||
// ============================================
|
||||
// Extension
|
||||
// ============================================
|
||||
|
||||
export default async function (pi: ExtensionAPI) {
|
||||
if (!state.config) {
|
||||
state.config = loadConfig();
|
||||
}
|
||||
|
||||
if (state.config) {
|
||||
console.log("📊 Langfuse: Tracing enabled →", state.config.host);
|
||||
} else {
|
||||
console.log("📊 Langfuse: Waiting for first-run setup");
|
||||
}
|
||||
|
||||
pi.registerCommand("langfuse-setup", {
|
||||
description: "Configure Langfuse API keys for this extension",
|
||||
handler: async (_args, ctx) => {
|
||||
await promptForConfig(ctx);
|
||||
},
|
||||
});
|
||||
|
||||
pi.registerCommand("langfuse-test", {
|
||||
description: "Send a test trace to Langfuse to verify configuration",
|
||||
handler: async (args, ctx) => {
|
||||
await handleLangfuseTestCommand(String(args ?? ""), ctx);
|
||||
},
|
||||
});
|
||||
|
||||
pi.registerCommand("langfuse-status", {
|
||||
description: "Show Langfuse configuration and runtime status",
|
||||
handler: async (args, ctx) => {
|
||||
await handleLangfuseStatusCommand(String(args ?? ""), ctx);
|
||||
},
|
||||
});
|
||||
|
||||
pi.registerCommand("langfuse-privacy", {
|
||||
description: "View or set Langfuse telemetry privacy preset",
|
||||
handler: async (args, ctx) => {
|
||||
await handleLangfusePrivacyCommand(String(args ?? ""), ctx);
|
||||
},
|
||||
});
|
||||
|
||||
const getSessionId = (ctx?: any) => {
|
||||
try {
|
||||
const sessionFile = ctx?.sessionManager?.getSessionFile?.();
|
||||
return sessionFile ? basename(sessionFile, ".jsonl") : undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
|
||||
const withSession = <T>(ctx: any, fn: () => T): T => runWithSession(getSessionId(ctx) ?? state.currentSessionId, fn);
|
||||
|
||||
pi.on("session_start", async (_event, ctx) => withSession(ctx, async () => {
|
||||
state.setupAttemptedThisSession = false;
|
||||
await ensureConfig(ctx);
|
||||
resetRunState();
|
||||
}));
|
||||
|
||||
pi.on("model_select", async (event, ctx) => withSession(ctx, async () => {
|
||||
state.currentModel = event.model?.id || "";
|
||||
state.currentProvider = event.model?.provider || "";
|
||||
}));
|
||||
|
||||
pi.on("before_agent_start", async (event, ctx) => withSession(ctx, async () => {
|
||||
await startAgentRun(event, ctx);
|
||||
}));
|
||||
|
||||
pi.on("agent_start", async (event, ctx) => withSession(ctx, async () => {
|
||||
if (!state.agentState?.root) {
|
||||
await startAgentRun(event, ctx);
|
||||
}
|
||||
}));
|
||||
|
||||
pi.on("turn_start", async (event, ctx) => withSession(ctx, async () => {
|
||||
await startTurnObservation(event);
|
||||
}));
|
||||
|
||||
pi.on("before_provider_request", async (event, ctx) => withSession(ctx, async () => {
|
||||
await startGeneration(event);
|
||||
}));
|
||||
|
||||
pi.on("after_provider_response", async (event, ctx) => withSession(ctx, async () => {
|
||||
updateGenerationMetadata(event);
|
||||
}));
|
||||
|
||||
pi.on("message_update", async (event, ctx) => withSession(ctx, async () => {
|
||||
recordTTFT(event);
|
||||
const message = getMessageFromEvent(event);
|
||||
if (message?.role === "assistant" && state.agentState) {
|
||||
state.agentState.latestAssistantOutput = extractAssistantOutput(message);
|
||||
}
|
||||
}));
|
||||
|
||||
pi.on("message_end", async (event, ctx) => withSession(ctx, async () => {
|
||||
await finishGenerationFromMessage(event);
|
||||
}));
|
||||
|
||||
pi.on("tool_execution_start", async (event, ctx) => withSession(ctx, async () => {
|
||||
await startToolObservation(event);
|
||||
}));
|
||||
|
||||
pi.on("tool_call", async (event, ctx) => withSession(ctx, async () => {
|
||||
await startToolObservation(event);
|
||||
}));
|
||||
|
||||
pi.on("tool_result", async (event, ctx) => withSession(ctx, async () => {
|
||||
await finishToolObservation(event);
|
||||
}));
|
||||
|
||||
pi.on("tool_execution_end", async (event, ctx) => withSession(ctx, async () => {
|
||||
await finishToolObservation(event);
|
||||
}));
|
||||
|
||||
pi.on("turn_end", async (event, ctx) => withSession(ctx, async () => {
|
||||
state.turnCount++;
|
||||
const message = getMessageFromEvent(event);
|
||||
if (message?.role === "assistant") {
|
||||
await createFallbackGenerationFromTurn(event, message);
|
||||
await finishGenerationFromMessage(event);
|
||||
}
|
||||
finishTurnObservation(event);
|
||||
}));
|
||||
|
||||
pi.on("agent_end", async (event, ctx) => withSession(ctx, async () => {
|
||||
await finishAgentRun(event);
|
||||
const sessionId = state.currentSessionId;
|
||||
setTimeout(() => {
|
||||
shutdownRuntime(sessionId).catch((error) => {
|
||||
console.warn("📊 Langfuse: Deferred shutdown failed", error);
|
||||
});
|
||||
}, 0);
|
||||
}));
|
||||
|
||||
const handleSessionInterruption = (reason: string) => {
|
||||
if (state.agentState?.root) {
|
||||
closeDanglingObservations(reason);
|
||||
state.agentState.root.update({ metadata: { completed: false, cancelled: true } }).end();
|
||||
}
|
||||
resetRunState();
|
||||
};
|
||||
|
||||
pi.on("session_before_switch", async (_event, ctx) => {
|
||||
const sessionId = getSessionId(ctx);
|
||||
if (sessionId) {
|
||||
setCurrentSession(sessionId);
|
||||
}
|
||||
});
|
||||
|
||||
pi.on("session_before_fork", async (_event, ctx) => {
|
||||
const sessionId = getSessionId(ctx);
|
||||
if (sessionId) {
|
||||
setCurrentSession(sessionId);
|
||||
}
|
||||
});
|
||||
|
||||
pi.on("session_compact", async (event, ctx) => withSession(ctx, async () => {
|
||||
if (state.agentState?.root) {
|
||||
const parent = state.agentState.activeTurn ?? state.agentState.root;
|
||||
try {
|
||||
const observation = parent.startObservation ? parent.startObservation(
|
||||
"session_compact",
|
||||
{
|
||||
level: "DEFAULT",
|
||||
statusMessage: "Context was compacted",
|
||||
metadata: applyCapturePolicy({ metadata: { ...event } }, getCapturePolicy()).metadata
|
||||
},
|
||||
{ asType: "span" }
|
||||
) : undefined;
|
||||
observation?.end();
|
||||
} catch (e) {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
}));
|
||||
|
||||
pi.on("session_shutdown", async (_event, ctx) => withSession(ctx, async () => {
|
||||
handleSessionInterruption("Session shutdown before agent completed");
|
||||
await shutdownRuntime();
|
||||
}));
|
||||
}
|
||||
5074
extensions/pi-langfuse/package-lock.json
generated
Normal file
5074
extensions/pi-langfuse/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
69
extensions/pi-langfuse/package.json
Normal file
69
extensions/pi-langfuse/package.json
Normal file
@@ -0,0 +1,69 @@
|
||||
{
|
||||
"name": "pi-langfuse",
|
||||
"version": "1.5.8",
|
||||
"description": "Langfuse extension for Pi coding agent",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/gooyoung/pi-langfuse.git"
|
||||
},
|
||||
"bugs": {
|
||||
"url": "https://github.com/gooyoung/pi-langfuse/issues"
|
||||
},
|
||||
"homepage": "https://github.com/gooyoung/pi-langfuse#readme",
|
||||
"type": "module",
|
||||
"packageManager": "npm@11.12.1",
|
||||
"main": "index.ts",
|
||||
"files": [
|
||||
"index.ts",
|
||||
"src/",
|
||||
"types/",
|
||||
"README.md",
|
||||
"README_CN.md",
|
||||
"image.png",
|
||||
"skills-lock.json",
|
||||
"tsconfig.json"
|
||||
],
|
||||
"scripts": {
|
||||
"typecheck": "tsc --noEmit",
|
||||
"test": "tsx --test test/*.test.ts"
|
||||
},
|
||||
"keywords": [
|
||||
"pi-package",
|
||||
"langfuse",
|
||||
"observability",
|
||||
"tracing",
|
||||
"monitoring",
|
||||
"pi-coding-agent",
|
||||
"extension"
|
||||
],
|
||||
"pi": {
|
||||
"extensions": [
|
||||
"./index.ts"
|
||||
],
|
||||
"image": "https://github.com/gooyoung/pi-langfuse/blob/main/image.png?raw=true"
|
||||
},
|
||||
"dependencies": {
|
||||
"@langfuse/client": "^5.3.0",
|
||||
"@langfuse/otel": "^5.3.0",
|
||||
"@langfuse/tracing": "^5.3.0",
|
||||
"@opentelemetry/api": "^1.9.0",
|
||||
"@opentelemetry/context-async-hooks": "^2.7.1",
|
||||
"@opentelemetry/sdk-node": "^0.218.0",
|
||||
"@opentelemetry/sdk-trace-base": "^2.0.1"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@earendil-works/pi-coding-agent": "*"
|
||||
},
|
||||
"publishConfig": {
|
||||
"access": "public",
|
||||
"registry": "https://registry.npmjs.org"
|
||||
},
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=22"
|
||||
},
|
||||
"devDependencies": {
|
||||
"tsx": "^4.19.0",
|
||||
"typescript": "^6.0.3"
|
||||
}
|
||||
}
|
||||
11
extensions/pi-langfuse/skills-lock.json
Normal file
11
extensions/pi-langfuse/skills-lock.json
Normal file
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"version": 1,
|
||||
"skills": {
|
||||
"langfuse": {
|
||||
"source": "langfuse/skills",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/langfuse/SKILL.md",
|
||||
"computedHash": "ccb3e0bee034850742b4387983e83c9cf2d8d8283a5a4264f0ecdfd03db01755"
|
||||
}
|
||||
}
|
||||
}
|
||||
141
extensions/pi-langfuse/src/capture-policy.ts
Normal file
141
extensions/pi-langfuse/src/capture-policy.ts
Normal file
@@ -0,0 +1,141 @@
|
||||
import { redactValue } from "./redaction.js";
|
||||
|
||||
export interface CapturePolicy {
|
||||
readonly captureInputs: boolean;
|
||||
readonly captureOutputs: boolean;
|
||||
readonly captureToolIo: boolean;
|
||||
readonly captureSystemPrompt: boolean;
|
||||
readonly captureCwd: boolean;
|
||||
}
|
||||
|
||||
export type PrivacyPreset = "metadata-only" | "prompts-only" | "conversations" | "full-debug";
|
||||
export type EnvLike = Readonly<Record<string, string | undefined>>;
|
||||
|
||||
export interface RawTelemetryPayload {
|
||||
input?: unknown;
|
||||
output?: unknown;
|
||||
toolInput?: unknown;
|
||||
toolOutput?: unknown;
|
||||
systemPrompt?: unknown;
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface CapturedTelemetryPayload {
|
||||
input?: unknown;
|
||||
output?: unknown;
|
||||
toolInput?: unknown;
|
||||
toolOutput?: unknown;
|
||||
systemPrompt?: unknown;
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
const PRESETS: Record<PrivacyPreset, CapturePolicy> = {
|
||||
"metadata-only": {
|
||||
captureInputs: false,
|
||||
captureOutputs: false,
|
||||
captureToolIo: false,
|
||||
captureSystemPrompt: false,
|
||||
captureCwd: false,
|
||||
},
|
||||
"prompts-only": {
|
||||
captureInputs: true,
|
||||
captureOutputs: false,
|
||||
captureToolIo: false,
|
||||
captureSystemPrompt: false,
|
||||
captureCwd: false,
|
||||
},
|
||||
conversations: {
|
||||
captureInputs: true,
|
||||
captureOutputs: true,
|
||||
captureToolIo: false,
|
||||
captureSystemPrompt: false,
|
||||
captureCwd: false,
|
||||
},
|
||||
"full-debug": {
|
||||
captureInputs: true,
|
||||
captureOutputs: true,
|
||||
captureToolIo: true,
|
||||
captureSystemPrompt: true,
|
||||
captureCwd: true,
|
||||
},
|
||||
};
|
||||
|
||||
const FLAG_TO_FIELD = {
|
||||
LANGFUSE_CAPTURE_INPUTS: "captureInputs",
|
||||
LANGFUSE_CAPTURE_OUTPUTS: "captureOutputs",
|
||||
LANGFUSE_CAPTURE_TOOL_IO: "captureToolIo",
|
||||
LANGFUSE_CAPTURE_SYSTEM_PROMPT: "captureSystemPrompt",
|
||||
LANGFUSE_CAPTURE_CWD: "captureCwd",
|
||||
} as const;
|
||||
|
||||
function parseFlag(value: string | undefined): boolean | undefined {
|
||||
if (value === undefined) {
|
||||
return undefined;
|
||||
}
|
||||
if (/^(1|true|yes|on)$/i.test(value)) {
|
||||
return true;
|
||||
}
|
||||
if (/^(0|false|no|off)$/i.test(value)) {
|
||||
return false;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function normalizePreset(value: string | undefined): PrivacyPreset {
|
||||
return value && value in PRESETS ? (value as PrivacyPreset) : "full-debug";
|
||||
}
|
||||
|
||||
export function createCapturePolicy(env: EnvLike = process.env as EnvLike): CapturePolicy {
|
||||
const policy: CapturePolicy = { ...PRESETS[normalizePreset(env.LANGFUSE_PRIVACY_PRESET)] };
|
||||
for (const [envName, field] of Object.entries(FLAG_TO_FIELD) as Array<
|
||||
[keyof typeof FLAG_TO_FIELD, (typeof FLAG_TO_FIELD)[keyof typeof FLAG_TO_FIELD]]
|
||||
>) {
|
||||
const override = parseFlag(env[envName]);
|
||||
if (override !== undefined) {
|
||||
(policy as Record<typeof field, boolean>)[field] = override;
|
||||
}
|
||||
}
|
||||
return policy;
|
||||
}
|
||||
|
||||
function redactMetadata(metadata: Record<string, unknown> | undefined, policy: CapturePolicy) {
|
||||
if (!metadata) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const output: Record<string, unknown> = {};
|
||||
for (const [key, value] of Object.entries(metadata)) {
|
||||
if (key === "cwd" && !policy.captureCwd) {
|
||||
continue;
|
||||
}
|
||||
output[key] = redactValue(value);
|
||||
}
|
||||
return Object.keys(output).length > 0 ? output : undefined;
|
||||
}
|
||||
|
||||
export function applyCapturePolicy(
|
||||
payload: RawTelemetryPayload,
|
||||
policy: CapturePolicy = createCapturePolicy(),
|
||||
): CapturedTelemetryPayload {
|
||||
const captured: CapturedTelemetryPayload = {
|
||||
metadata: redactMetadata(payload.metadata, policy),
|
||||
};
|
||||
|
||||
if (policy.captureInputs && "input" in payload) {
|
||||
captured.input = redactValue(payload.input);
|
||||
}
|
||||
if (policy.captureOutputs && "output" in payload) {
|
||||
captured.output = redactValue(payload.output);
|
||||
}
|
||||
if (policy.captureToolIo && "toolInput" in payload) {
|
||||
captured.toolInput = redactValue(payload.toolInput);
|
||||
}
|
||||
if (policy.captureToolIo && "toolOutput" in payload) {
|
||||
captured.toolOutput = redactValue(payload.toolOutput);
|
||||
}
|
||||
if (policy.captureSystemPrompt && "systemPrompt" in payload) {
|
||||
captured.systemPrompt = redactValue(payload.systemPrompt);
|
||||
}
|
||||
|
||||
return captured;
|
||||
}
|
||||
423
extensions/pi-langfuse/src/commands.ts
Normal file
423
extensions/pi-langfuse/src/commands.ts
Normal file
@@ -0,0 +1,423 @@
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
|
||||
import { CONFIG_PATH } from "./constants.js";
|
||||
import {
|
||||
loadConfig,
|
||||
loadConfigFromEnv,
|
||||
loadConfigFromFile,
|
||||
sanitizeConfigForLog,
|
||||
saveConfig,
|
||||
ensureConfig,
|
||||
} from "./config.js";
|
||||
import { createCapturePolicy, type PrivacyPreset, type CapturePolicy } from "./capture-policy.js";
|
||||
import { getRuntime, getLastRuntimeError, forceShutdownRuntime as shutdownLangfuseRuntime } from "./langfuse.js";
|
||||
import { state } from "./state.js";
|
||||
import type { Config, LangfuseRuntime } from "./types.js";
|
||||
|
||||
const PRIVACY_PRESETS = ["metadata-only", "prompts-only", "conversations", "full-debug"] as const;
|
||||
|
||||
export interface CommandContextLike {
|
||||
hasUI?: boolean;
|
||||
ui?: {
|
||||
notify?: (message: string, level?: "info" | "warning" | "error") => void;
|
||||
select?: (title: string, options: string[]) => Promise<string | undefined>;
|
||||
};
|
||||
}
|
||||
|
||||
interface CommandDeps {
|
||||
configPath?: string;
|
||||
getRuntime?: () => Promise<LangfuseRuntime>;
|
||||
forceShutdownRuntime?: () => Promise<void>;
|
||||
env?: Record<string, string | undefined>;
|
||||
checkConnectivity?: (config: Config) => Promise<ConnectivityResult>;
|
||||
}
|
||||
|
||||
interface ConnectivityResult {
|
||||
ok: boolean;
|
||||
message: string;
|
||||
}
|
||||
|
||||
function notify(ctx: CommandContextLike, message: string, level: "info" | "warning" | "error" = "info") {
|
||||
if (ctx.hasUI && ctx.ui?.notify) {
|
||||
ctx.ui.notify(message, level);
|
||||
return;
|
||||
}
|
||||
const prefix = level === "error" ? "❌" : level === "warning" ? "⚠️" : "📊";
|
||||
console.log(`${prefix} Langfuse: ${message}`);
|
||||
}
|
||||
|
||||
function parseCommandArgs(args: string): { values: Record<string, string>; positional: string[]; malformed: string[] } {
|
||||
const values: Record<string, string> = {};
|
||||
const positional: string[] = [];
|
||||
const malformed: string[] = [];
|
||||
|
||||
for (const part of args.trim().split(/\s+/)) {
|
||||
if (!part) {
|
||||
continue;
|
||||
}
|
||||
const eq = part.indexOf("=");
|
||||
if (eq === -1) {
|
||||
positional.push(part);
|
||||
continue;
|
||||
}
|
||||
if (eq === 0) {
|
||||
malformed.push(part);
|
||||
continue;
|
||||
}
|
||||
values[part.slice(0, eq)] = part.slice(eq + 1);
|
||||
}
|
||||
|
||||
return { values, positional, malformed };
|
||||
}
|
||||
|
||||
function isPrivacyPreset(value: string | undefined): value is PrivacyPreset {
|
||||
return PRIVACY_PRESETS.includes(value as PrivacyPreset);
|
||||
}
|
||||
|
||||
function inferPreset(policy: CapturePolicy): PrivacyPreset | "custom" {
|
||||
const entries: Array<[PrivacyPreset, CapturePolicy]> = [
|
||||
[
|
||||
"metadata-only",
|
||||
{
|
||||
captureInputs: false,
|
||||
captureOutputs: false,
|
||||
captureToolIo: false,
|
||||
captureSystemPrompt: false,
|
||||
captureCwd: false,
|
||||
},
|
||||
],
|
||||
[
|
||||
"prompts-only",
|
||||
{
|
||||
captureInputs: true,
|
||||
captureOutputs: false,
|
||||
captureToolIo: false,
|
||||
captureSystemPrompt: false,
|
||||
captureCwd: false,
|
||||
},
|
||||
],
|
||||
[
|
||||
"conversations",
|
||||
{
|
||||
captureInputs: true,
|
||||
captureOutputs: true,
|
||||
captureToolIo: false,
|
||||
captureSystemPrompt: false,
|
||||
captureCwd: false,
|
||||
},
|
||||
],
|
||||
[
|
||||
"full-debug",
|
||||
{
|
||||
captureInputs: true,
|
||||
captureOutputs: true,
|
||||
captureToolIo: true,
|
||||
captureSystemPrompt: true,
|
||||
captureCwd: true,
|
||||
},
|
||||
],
|
||||
];
|
||||
|
||||
for (const [preset, presetPolicy] of entries) {
|
||||
if (
|
||||
policy.captureInputs === presetPolicy.captureInputs &&
|
||||
policy.captureOutputs === presetPolicy.captureOutputs &&
|
||||
policy.captureToolIo === presetPolicy.captureToolIo &&
|
||||
policy.captureSystemPrompt === presetPolicy.captureSystemPrompt &&
|
||||
policy.captureCwd === presetPolicy.captureCwd
|
||||
) {
|
||||
return preset;
|
||||
}
|
||||
}
|
||||
return "custom";
|
||||
}
|
||||
|
||||
function describePolicy(policy: CapturePolicy) {
|
||||
return [
|
||||
`captureInputs: ${policy.captureInputs}`,
|
||||
`captureOutputs: ${policy.captureOutputs}`,
|
||||
`captureToolIo: ${policy.captureToolIo}`,
|
||||
`captureSystemPrompt: ${policy.captureSystemPrompt}`,
|
||||
`captureCwd: ${policy.captureCwd}`,
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function flag(value: boolean): "on" | "off" {
|
||||
return value ? "on" : "off";
|
||||
}
|
||||
|
||||
function readPersistedConfig(path: string) {
|
||||
if (!existsSync(path)) {
|
||||
return {};
|
||||
}
|
||||
try {
|
||||
return JSON.parse(readFileSync(path, "utf-8")) as Record<string, unknown>;
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
function hasActiveAgentObservation() {
|
||||
for (const sessionState of state.sessionStates.values()) {
|
||||
if (sessionState.agentState?.root) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function configSource(env: Record<string, string | undefined>, configPath: string): string {
|
||||
const fileConfig = loadConfigFromFile(configPath, env);
|
||||
const envConfig = loadConfigFromEnv(env);
|
||||
if (fileConfig && envConfig) {
|
||||
return "config file (env capture flags may override saved privacy)";
|
||||
}
|
||||
if (fileConfig) {
|
||||
return "config file";
|
||||
}
|
||||
if (envConfig) {
|
||||
return "environment variables";
|
||||
}
|
||||
return "none";
|
||||
}
|
||||
|
||||
function lastErrorSummary() {
|
||||
const lastError = getLastRuntimeError();
|
||||
if (!lastError) {
|
||||
return "none";
|
||||
}
|
||||
return `${lastError.scope}: ${lastError.message} (${lastError.timestamp.toISOString()})`;
|
||||
}
|
||||
|
||||
function formatStatus(configPath: string, env: Record<string, string | undefined>) {
|
||||
const config = loadConfig(env, configPath);
|
||||
if (!config) {
|
||||
return [
|
||||
"pi-langfuse status:",
|
||||
"State: not configured",
|
||||
`Config file: ${configPath}`,
|
||||
"Action: run /langfuse-setup or set LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY",
|
||||
`Last error: ${lastErrorSummary()}`,
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
const safeConfig = sanitizeConfigForLog(config);
|
||||
const policy = config.capturePolicy ?? createCapturePolicy(env);
|
||||
return [
|
||||
"pi-langfuse status:",
|
||||
"State: configured",
|
||||
`Source: ${configSource(env, configPath)}`,
|
||||
`Host: ${safeConfig?.host ?? config.host}`,
|
||||
`Public key: ${safeConfig?.publicKey ?? "[REDACTED_SECRET]"}`,
|
||||
`Config file: ${configPath}`,
|
||||
`Privacy preset: ${inferPreset(policy)}`,
|
||||
"Capture:",
|
||||
` inputs: ${flag(policy.captureInputs)}`,
|
||||
` outputs: ${flag(policy.captureOutputs)}`,
|
||||
` tool IO: ${flag(policy.captureToolIo)}`,
|
||||
` system prompt: ${flag(policy.captureSystemPrompt)}`,
|
||||
` cwd: ${flag(policy.captureCwd)}`,
|
||||
`Active run: ${hasActiveAgentObservation() ? "yes" : "no"}`,
|
||||
`Last error: ${lastErrorSummary()}`,
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
async function checkLangfuseConnectivity(config: Config): Promise<ConnectivityResult> {
|
||||
const host = config.host.replace(/\/+$/, "");
|
||||
const auth = Buffer.from(`${config.publicKey}:${config.secretKey}`).toString("base64");
|
||||
|
||||
try {
|
||||
const response = await fetch(`${host}/api/public/projects`, {
|
||||
headers: {
|
||||
Authorization: `Basic ${auth}`,
|
||||
},
|
||||
signal: AbortSignal.timeout(10_000),
|
||||
});
|
||||
|
||||
if (response.ok) {
|
||||
return { ok: true, message: `Connected to ${config.host}` };
|
||||
}
|
||||
|
||||
return {
|
||||
ok: false,
|
||||
message: `${config.host} returned ${response.status} ${response.statusText}`.trim(),
|
||||
};
|
||||
} catch (error) {
|
||||
return {
|
||||
ok: false,
|
||||
message: error instanceof Error ? error.message : String(error),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
export async function handleLangfuseStatusCommand(
|
||||
args: string,
|
||||
ctx: CommandContextLike,
|
||||
deps: CommandDeps = {},
|
||||
): Promise<boolean> {
|
||||
const parsed = parseCommandArgs(args);
|
||||
const unexpected = parsed.malformed[0] ?? parsed.positional[0] ?? Object.keys(parsed.values)[0];
|
||||
if (unexpected) {
|
||||
notify(ctx, `Unexpected argument '${unexpected}'. Usage: /langfuse-status`, "warning");
|
||||
return false;
|
||||
}
|
||||
|
||||
const env = deps.env ?? process.env;
|
||||
const configPath = deps.configPath ?? CONFIG_PATH;
|
||||
notify(ctx, formatStatus(configPath, env));
|
||||
return true;
|
||||
}
|
||||
|
||||
function savePrivacyPreset(
|
||||
requestedPreset: PrivacyPreset,
|
||||
ctx: CommandContextLike,
|
||||
configPath: string,
|
||||
): boolean {
|
||||
const existing = readPersistedConfig(configPath);
|
||||
const loaded = state.config ?? loadConfig(process.env, configPath);
|
||||
|
||||
const publicKey = existing.publicKey ?? loaded?.publicKey;
|
||||
const secretKey = existing.secretKey ?? loaded?.secretKey;
|
||||
const host = existing.host ?? loaded?.host;
|
||||
|
||||
if (!publicKey || !secretKey || !host) {
|
||||
notify(ctx, "Langfuse is not configured yet. Run /langfuse-setup before changing privacy settings.", "warning");
|
||||
return false;
|
||||
}
|
||||
|
||||
const nextConfig = {
|
||||
publicKey: String(publicKey),
|
||||
secretKey: String(secretKey),
|
||||
host: String(host),
|
||||
privacyPreset: requestedPreset,
|
||||
};
|
||||
saveConfig(nextConfig, configPath);
|
||||
state.config = loadConfig(process.env, configPath);
|
||||
|
||||
notify(ctx, `Langfuse privacy preset saved: ${requestedPreset}\n${describePolicy(state.config?.capturePolicy ?? createCapturePolicy())}`);
|
||||
return true;
|
||||
}
|
||||
|
||||
export async function handleLangfusePrivacyCommand(
|
||||
args: string,
|
||||
ctx: CommandContextLike,
|
||||
deps: CommandDeps = {},
|
||||
): Promise<boolean> {
|
||||
const configPath = deps.configPath ?? CONFIG_PATH;
|
||||
const parsed = parseCommandArgs(args);
|
||||
if (parsed.malformed.length > 0) {
|
||||
notify(ctx, `Couldn't understand '${parsed.malformed[0]}'. Use /langfuse-privacy preset=metadata-only.`, "warning");
|
||||
return false;
|
||||
}
|
||||
|
||||
const requestedPreset = parsed.values.preset ?? parsed.positional[0];
|
||||
if (!requestedPreset) {
|
||||
state.config = state.config ?? loadConfig(process.env, configPath);
|
||||
const policy = state.config?.capturePolicy ?? createCapturePolicy();
|
||||
if (ctx.hasUI && ctx.ui?.select) {
|
||||
const currentPreset = inferPreset(policy);
|
||||
const selectedPreset = await ctx.ui.select(
|
||||
`Langfuse privacy preset (current: ${currentPreset})`,
|
||||
[...PRIVACY_PRESETS],
|
||||
);
|
||||
if (!selectedPreset) {
|
||||
notify(ctx, `Current Langfuse privacy preset: ${currentPreset}\n${describePolicy(policy)}`);
|
||||
return true;
|
||||
}
|
||||
if (!isPrivacyPreset(selectedPreset)) {
|
||||
notify(ctx, `Unknown privacy preset '${selectedPreset}'. Use one of: ${PRIVACY_PRESETS.join(", ")}.`, "warning");
|
||||
return false;
|
||||
}
|
||||
return savePrivacyPreset(selectedPreset, ctx, configPath);
|
||||
}
|
||||
notify(ctx, `Current Langfuse privacy preset: ${inferPreset(policy)}\n${describePolicy(policy)}`);
|
||||
return true;
|
||||
}
|
||||
|
||||
if (!isPrivacyPreset(requestedPreset)) {
|
||||
notify(
|
||||
ctx,
|
||||
`Unknown privacy preset '${requestedPreset}'. Use one of: ${PRIVACY_PRESETS.join(", ")}.`,
|
||||
"warning",
|
||||
);
|
||||
return false;
|
||||
}
|
||||
|
||||
return savePrivacyPreset(requestedPreset, ctx, configPath);
|
||||
}
|
||||
|
||||
export async function handleLangfuseTestCommand(
|
||||
args: string,
|
||||
ctx: CommandContextLike,
|
||||
deps: CommandDeps = {},
|
||||
): Promise<boolean> {
|
||||
const parsed = parseCommandArgs(args);
|
||||
const unexpected = parsed.malformed[0] ?? parsed.positional[0] ?? Object.keys(parsed.values)[0];
|
||||
if (unexpected) {
|
||||
notify(ctx, `Unexpected argument '${unexpected}'. Usage: /langfuse-test`, "warning");
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!state.config && !(await ensureConfig(ctx))) {
|
||||
notify(ctx, "Langfuse is not configured yet. Run /langfuse-setup first.", "warning");
|
||||
return false;
|
||||
}
|
||||
|
||||
if (hasActiveAgentObservation()) {
|
||||
notify(ctx, "Langfuse test skipped because an agent run is active. Try again after the run finishes.", "warning");
|
||||
return false;
|
||||
}
|
||||
|
||||
const config = state.config;
|
||||
if (!config) {
|
||||
notify(ctx, "Langfuse is not configured yet. Run /langfuse-setup first.", "warning");
|
||||
return false;
|
||||
}
|
||||
|
||||
const connectivity = await (deps.checkConnectivity ?? checkLangfuseConnectivity)(config);
|
||||
if (!connectivity.ok) {
|
||||
notify(ctx, `Langfuse connectivity check failed: ${connectivity.message}`, "error");
|
||||
return false;
|
||||
}
|
||||
|
||||
let runtimeInitialized = false;
|
||||
try {
|
||||
const rt = await (deps.getRuntime ?? getRuntime)();
|
||||
runtimeInitialized = true;
|
||||
rt.propagateAttributes(
|
||||
{
|
||||
traceName: "pi-langfuse-test",
|
||||
metadata: {
|
||||
source: "pi-langfuse",
|
||||
command: "langfuse-test",
|
||||
},
|
||||
},
|
||||
() => {
|
||||
const observation = rt.startObservation(
|
||||
"pi-langfuse-test",
|
||||
{
|
||||
input: { command: "/langfuse-test" },
|
||||
output: "ok",
|
||||
metadata: {
|
||||
source: "pi-langfuse",
|
||||
command: "langfuse-test",
|
||||
},
|
||||
},
|
||||
{ asType: "span" },
|
||||
);
|
||||
observation.end();
|
||||
return observation;
|
||||
},
|
||||
);
|
||||
notify(ctx, `Langfuse test succeeded. ${connectivity.message}; test trace sent to ${config.host}.`);
|
||||
return true;
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
notify(ctx, `Langfuse test failed: ${message}`, "error");
|
||||
return false;
|
||||
} finally {
|
||||
if (runtimeInitialized) {
|
||||
await (deps.forceShutdownRuntime ?? shutdownLangfuseRuntime)();
|
||||
}
|
||||
}
|
||||
}
|
||||
169
extensions/pi-langfuse/src/config.ts
Normal file
169
extensions/pi-langfuse/src/config.ts
Normal file
@@ -0,0 +1,169 @@
|
||||
import { chmodSync, mkdirSync, readFileSync, existsSync, writeFileSync } from "node:fs";
|
||||
import { dirname } from "node:path";
|
||||
import type { Config } from "./types.js";
|
||||
import { CONFIG_PATH, DEFAULT_LANGFUSE_HOST } from "./constants.js";
|
||||
import { state } from "./state.js";
|
||||
import { forceShutdownRuntime } from "./langfuse.js";
|
||||
import { createCapturePolicy, type EnvLike } from "./capture-policy.js";
|
||||
import { createPayloadLimits } from "./limits.js";
|
||||
|
||||
export function loadConfigFromFile(path = CONFIG_PATH, env: EnvLike = process.env as EnvLike): Config | null {
|
||||
if (existsSync(path)) {
|
||||
try {
|
||||
const content = readFileSync(path, "utf-8");
|
||||
const config = JSON.parse(content) as Config & { capture?: EnvLike; privacyPreset?: string };
|
||||
if (config.publicKey && config.secretKey) {
|
||||
const captureSource: EnvLike = {
|
||||
...(config.capture ?? {}),
|
||||
...(config.privacyPreset ? { LANGFUSE_PRIVACY_PRESET: config.privacyPreset } : {}),
|
||||
...env,
|
||||
};
|
||||
return {
|
||||
publicKey: config.publicKey,
|
||||
secretKey: config.secretKey,
|
||||
host: config.host || DEFAULT_LANGFUSE_HOST,
|
||||
capturePolicy: createCapturePolicy(captureSource),
|
||||
limits: createPayloadLimits(env),
|
||||
};
|
||||
}
|
||||
} catch (e) {
|
||||
console.warn("📊 Langfuse: Failed to load config.json", e);
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
export function loadConfigFromEnv(env: EnvLike = process.env as EnvLike): Config | null {
|
||||
const publicKey = env.LANGFUSE_PUBLIC_KEY || "";
|
||||
const secretKey = env.LANGFUSE_SECRET_KEY || "";
|
||||
if (!publicKey || !secretKey) {
|
||||
return null;
|
||||
}
|
||||
|
||||
return {
|
||||
publicKey,
|
||||
secretKey,
|
||||
host: env.LANGFUSE_BASE_URL || env.LANGFUSE_HOST || DEFAULT_LANGFUSE_HOST,
|
||||
capturePolicy: createCapturePolicy(env),
|
||||
limits: createPayloadLimits(env),
|
||||
};
|
||||
}
|
||||
|
||||
export function loadConfig(env: EnvLike = process.env as EnvLike, path = CONFIG_PATH): Config | null {
|
||||
return loadConfigFromFile(path, env) || loadConfigFromEnv(env);
|
||||
}
|
||||
|
||||
export function saveConfig(config: Config, path = CONFIG_PATH) {
|
||||
mkdirSync(dirname(path), { recursive: true, mode: 0o700 });
|
||||
chmodSync(dirname(path), 0o700);
|
||||
writeFileSync(path, `${JSON.stringify(config, null, 2)}\n`, { encoding: "utf-8", mode: 0o600 });
|
||||
chmodSync(path, 0o600);
|
||||
}
|
||||
|
||||
function maskPublicKey(value: string): string {
|
||||
if (value.length <= 9) {
|
||||
return "[REDACTED_SECRET]";
|
||||
}
|
||||
return `${value.slice(0, 6)}...${value.slice(-4)}`;
|
||||
}
|
||||
|
||||
export function sanitizeConfigForLog(config: Pick<Config, "publicKey" | "secretKey" | "host"> | null): {
|
||||
publicKey: string;
|
||||
secretKey: string;
|
||||
host: string;
|
||||
} | null {
|
||||
if (!config) {
|
||||
return null;
|
||||
}
|
||||
|
||||
return {
|
||||
publicKey: maskPublicKey(config.publicKey),
|
||||
secretKey: "[REDACTED_SECRET]",
|
||||
host: config.host || DEFAULT_LANGFUSE_HOST,
|
||||
};
|
||||
}
|
||||
|
||||
async function collectConfigFromUI(ctx: any, reason: string): Promise<Config | null> {
|
||||
if (!ctx.hasUI) {
|
||||
console.log(`📊 Langfuse: ${reason}. Run this extension in Pi UI to complete setup, or set LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY / LANGFUSE_BASE_URL.`);
|
||||
return null;
|
||||
}
|
||||
|
||||
ctx.ui.notify("Langfuse setup required. Enter your API keys to enable tracing.", "info");
|
||||
|
||||
const publicKey = (await ctx.ui.input("Langfuse public key:", "pk-lf-..."))?.trim();
|
||||
if (!publicKey) {
|
||||
ctx.ui.notify("Langfuse setup cancelled.", "warning");
|
||||
return null;
|
||||
}
|
||||
|
||||
const secretKey = (await ctx.ui.input("Langfuse secret key:", "sk-lf-..."))?.trim();
|
||||
if (!secretKey) {
|
||||
ctx.ui.notify("Langfuse setup cancelled.", "warning");
|
||||
return null;
|
||||
}
|
||||
|
||||
const hostInput = (await ctx.ui.input("Langfuse host:", DEFAULT_LANGFUSE_HOST))?.trim();
|
||||
return {
|
||||
publicKey,
|
||||
secretKey,
|
||||
host: hostInput || DEFAULT_LANGFUSE_HOST,
|
||||
capturePolicy: createCapturePolicy(),
|
||||
};
|
||||
}
|
||||
|
||||
async function saveConfigFromUI(ctx: any, config: Config): Promise<boolean> {
|
||||
state.config = config;
|
||||
|
||||
try {
|
||||
saveConfig(state.config);
|
||||
ctx.ui.notify(`Langfuse config saved to ${CONFIG_PATH}`, "info");
|
||||
return true;
|
||||
} catch (error) {
|
||||
console.warn("📊 Langfuse: Failed to save config.json", error);
|
||||
ctx.ui.notify(`Failed to save Langfuse config.json to ${CONFIG_PATH}. Check Pi config directory permissions.`, "error");
|
||||
state.config = null;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export async function ensureConfig(ctx: any): Promise<boolean> {
|
||||
if (!state.config) {
|
||||
state.config = loadConfig();
|
||||
}
|
||||
|
||||
if (state.config) {
|
||||
return true;
|
||||
}
|
||||
|
||||
if (state.setupAttemptedThisSession) {
|
||||
return false;
|
||||
}
|
||||
state.setupAttemptedThisSession = true;
|
||||
|
||||
const config = await collectConfigFromUI(ctx, "Missing config");
|
||||
if (!config) {
|
||||
return false;
|
||||
}
|
||||
|
||||
return saveConfigFromUI(ctx, config);
|
||||
}
|
||||
|
||||
export async function promptForConfig(ctx: any): Promise<boolean> {
|
||||
state.setupAttemptedThisSession = false;
|
||||
state.config = null;
|
||||
await forceShutdownRuntime();
|
||||
|
||||
const config = await collectConfigFromUI(ctx, "Manual setup requested");
|
||||
if (!config) {
|
||||
state.config = loadConfig();
|
||||
return false;
|
||||
}
|
||||
|
||||
const saved = await saveConfigFromUI(ctx, config);
|
||||
if (saved) {
|
||||
ctx.ui.notify("Langfuse tracing enabled for future agent runs.", "info");
|
||||
}
|
||||
return saved;
|
||||
}
|
||||
13
extensions/pi-langfuse/src/constants.ts
Normal file
13
extensions/pi-langfuse/src/constants.ts
Normal file
@@ -0,0 +1,13 @@
|
||||
import { homedir } from "node:os";
|
||||
import { resolve } from "node:path";
|
||||
|
||||
export const CONFIG_DIR = resolve(homedir(), ".pi", "agent", "pi-langfuse");
|
||||
export const CONFIG_PATH = resolve(CONFIG_DIR, "config.json");
|
||||
export const DEFAULT_LANGFUSE_HOST = "https://cloud.langfuse.com";
|
||||
|
||||
export const MAX_STRING_LENGTH = 12_000;
|
||||
export const MAX_TOOL_PAYLOAD_LENGTH = 24_000;
|
||||
export const MAX_DEPTH = 6;
|
||||
export const MAX_ARRAY_ITEMS = 50;
|
||||
export const MAX_OBJECT_KEYS = 80;
|
||||
export const MAX_PAYLOAD_NODES = 2_000;
|
||||
173
extensions/pi-langfuse/src/handlers/agent.ts
Normal file
173
extensions/pi-langfuse/src/handlers/agent.ts
Normal file
@@ -0,0 +1,173 @@
|
||||
import { state, resetRunState, computeEvaluationScores } from "../state.js";
|
||||
import { getRuntime, sendScore } from "../langfuse.js";
|
||||
import { ensureConfig } from "../config.js";
|
||||
import { shapePayload, truncate, extractFinalAssistant, extractAssistantOutput, getCapturePolicy, getLimits } from "../utils.js";
|
||||
import { closeDanglingObservations } from "./tool.js";
|
||||
import { applyCapturePolicy } from "../capture-policy.js";
|
||||
import { collectSourceMetadata } from "../source-metadata.js";
|
||||
|
||||
function stringMetadata(metadata: Record<string, unknown> | undefined): Record<string, string> | undefined {
|
||||
if (!metadata) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const output: Record<string, string> = {};
|
||||
for (const [key, value] of Object.entries(metadata)) {
|
||||
if (typeof value === "string") {
|
||||
output[key] = value;
|
||||
} else if (typeof value === "number" || typeof value === "boolean" || typeof value === "bigint") {
|
||||
output[key] = String(value);
|
||||
}
|
||||
}
|
||||
return Object.keys(output).length > 0 ? output : undefined;
|
||||
}
|
||||
|
||||
export function updateTraceIO(input?: unknown, output?: unknown) {
|
||||
const root = state.agentState?.root;
|
||||
if (!root?.setTraceIO) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
root.setTraceIO({ input, output });
|
||||
} catch {
|
||||
// Older SDKs may omit setTraceIO; root IO still mirrors trace IO in current Langfuse.
|
||||
}
|
||||
}
|
||||
|
||||
export async function startAgentRun(event: Record<string, unknown>, ctx: any) {
|
||||
if (!(await ensureConfig(ctx))) {
|
||||
state.isTracingDisabled = true;
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const rt = await getRuntime();
|
||||
const cwd = String(
|
||||
(event.systemPromptOptions && typeof event.systemPromptOptions === "object"
|
||||
? (event.systemPromptOptions as Record<string, unknown>).cwd
|
||||
: undefined) ?? process.cwd(),
|
||||
);
|
||||
|
||||
if (!state.currentModel && ctx.model) {
|
||||
state.currentModel = ctx.model.id || "";
|
||||
state.currentProvider = ctx.model.provider || "";
|
||||
}
|
||||
|
||||
let systemPrompt = undefined;
|
||||
try {
|
||||
if (ctx.getSystemPrompt) {
|
||||
systemPrompt = await ctx.getSystemPrompt();
|
||||
}
|
||||
} catch {
|
||||
// Ignore if getSystemPrompt is not available or fails
|
||||
}
|
||||
|
||||
const rawPromptInput = shapePayload({
|
||||
prompt: event.prompt,
|
||||
images: event.images,
|
||||
context: event.context ?? event.attachments,
|
||||
});
|
||||
const sourceMetadata = collectSourceMetadata(cwd);
|
||||
const captured = applyCapturePolicy(
|
||||
{
|
||||
input: rawPromptInput,
|
||||
metadata: {
|
||||
cwd,
|
||||
...sourceMetadata,
|
||||
...(state.currentModel ? { model: state.currentModel } : {}),
|
||||
...(state.currentProvider ? { provider: state.currentProvider } : {}),
|
||||
sessionId: state.currentSessionId || undefined,
|
||||
},
|
||||
systemPrompt: systemPrompt ? truncate(String(systemPrompt), getLimits().maxString) : undefined,
|
||||
},
|
||||
getCapturePolicy(),
|
||||
);
|
||||
|
||||
state.agentState = {
|
||||
cwd,
|
||||
promptInput: captured.input,
|
||||
generationSeq: 0,
|
||||
activeGenerations: new Map(),
|
||||
generationOrder: [],
|
||||
activeTools: new Map(),
|
||||
sourceMetadata,
|
||||
providerMetadataByRequest: new Map(),
|
||||
};
|
||||
|
||||
const root = rt.propagateAttributes(
|
||||
{
|
||||
sessionId: state.currentSessionId ? truncate(state.currentSessionId, 200) : undefined,
|
||||
traceName: "pi-agent",
|
||||
metadata: stringMetadata(captured.metadata),
|
||||
},
|
||||
() =>
|
||||
rt.startObservation(
|
||||
"pi-agent",
|
||||
{
|
||||
input: captured.input,
|
||||
metadata: {
|
||||
...(captured.metadata ?? {}),
|
||||
...(captured.systemPrompt ? { systemPrompt: captured.systemPrompt } : {}),
|
||||
},
|
||||
},
|
||||
{ asType: "agent" },
|
||||
),
|
||||
);
|
||||
|
||||
state.agentState.root = root;
|
||||
state.agentState.traceId = root.traceId;
|
||||
updateTraceIO(captured.input, undefined);
|
||||
} catch (e) {
|
||||
console.warn("📊 Langfuse: Failed to create agent observation", e);
|
||||
state.isTracingDisabled = true;
|
||||
}
|
||||
}
|
||||
|
||||
export async function finishAgentRun(event: Record<string, unknown> = {}) {
|
||||
if (!state.agentState?.root) {
|
||||
resetRunState();
|
||||
return;
|
||||
}
|
||||
|
||||
const lastAssistant = extractFinalAssistant(event.messages);
|
||||
const rawOutput = lastAssistant ? extractAssistantOutput(lastAssistant) : state.agentState.latestAssistantOutput;
|
||||
const captured = applyCapturePolicy(
|
||||
{
|
||||
output: rawOutput,
|
||||
metadata: {
|
||||
cwd: state.agentState.cwd,
|
||||
...(state.agentState.sourceMetadata ?? {}),
|
||||
completed: true,
|
||||
model: state.currentModel || undefined,
|
||||
provider: state.currentProvider || undefined,
|
||||
totalTools: state.toolCallCount,
|
||||
...computeEvaluationScores(),
|
||||
},
|
||||
},
|
||||
getCapturePolicy(),
|
||||
);
|
||||
const scores = computeEvaluationScores();
|
||||
|
||||
closeDanglingObservations("Agent run ended before observation finalized");
|
||||
|
||||
try {
|
||||
state.agentState.root
|
||||
.update({
|
||||
output: captured.output,
|
||||
metadata: captured.metadata,
|
||||
})
|
||||
.end();
|
||||
updateTraceIO(state.agentState.promptInput, captured.output);
|
||||
|
||||
await sendScore("tool_call_count", scores.tool_call_count, { traceId: state.agentState.traceId });
|
||||
await sendScore("turn_count", scores.turn_count, { traceId: state.agentState.traceId });
|
||||
await sendScore("total_tool_errors", scores.total_tool_errors, { traceId: state.agentState.traceId });
|
||||
await sendScore("tool_success_rate", scores.tool_success_rate, { traceId: state.agentState.traceId });
|
||||
await sendScore("session_had_errors", scores.session_had_errors, { traceId: state.agentState.traceId });
|
||||
} catch (e) {
|
||||
console.warn("📊 Langfuse: Failed to finish agent observation", e);
|
||||
} finally {
|
||||
resetRunState();
|
||||
}
|
||||
}
|
||||
246
extensions/pi-langfuse/src/handlers/generation.ts
Normal file
246
extensions/pi-langfuse/src/handlers/generation.ts
Normal file
@@ -0,0 +1,246 @@
|
||||
import { state } from "../state.js";
|
||||
import { getRuntime } from "../langfuse.js";
|
||||
import { startChildObservation } from "../observation.js";
|
||||
import {
|
||||
getRequestKey,
|
||||
getProviderPayload,
|
||||
shapePayload,
|
||||
extractResponseMetadata,
|
||||
getMessageFromEvent,
|
||||
extractAssistantOutput,
|
||||
extractUsage,
|
||||
extractCostDetails,
|
||||
getCapturePolicy,
|
||||
extractModelParameters,
|
||||
} from "../utils.js";
|
||||
import type { GenerationState, ObservationUpdate } from "../types.js";
|
||||
import { applyCapturePolicy } from "../capture-policy.js";
|
||||
|
||||
export function getOpenGeneration(): GenerationState | undefined {
|
||||
if (state.isTracingDisabled || !state.agentState) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
for (let i = state.agentState.generationOrder.length - 1; i >= 0; i--) {
|
||||
const key = state.agentState.generationOrder[i];
|
||||
const genState = state.agentState.activeGenerations.get(key);
|
||||
if (genState && !genState.ended) {
|
||||
return genState;
|
||||
}
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
export async function startGeneration(event: Record<string, unknown>) {
|
||||
if (state.isTracingDisabled || !state.agentState?.root) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const key = getRequestKey(event, `generation-${++state.agentState.generationSeq}`);
|
||||
const payload = getProviderPayload(event);
|
||||
const modelParameters = extractModelParameters(payload);
|
||||
const model = String(event.model ?? event.modelId ?? state.currentModel ?? "");
|
||||
const provider = String(event.provider ?? state.currentProvider ?? "");
|
||||
const metadata = shapePayload({
|
||||
provider,
|
||||
requestId: key,
|
||||
url: event.url,
|
||||
method: event.method,
|
||||
}) as Record<string, unknown>;
|
||||
const captured = applyCapturePolicy(
|
||||
{
|
||||
input: shapePayload(payload),
|
||||
metadata,
|
||||
},
|
||||
getCapturePolicy(),
|
||||
);
|
||||
|
||||
const parent = state.agentState.activeTurn ?? state.agentState.root;
|
||||
const generation = await startChildObservation({
|
||||
parent,
|
||||
runtime: getRuntime,
|
||||
name: "llm-generation",
|
||||
body: {
|
||||
input: captured.input,
|
||||
model: model || undefined,
|
||||
modelParameters,
|
||||
metadata: captured.metadata,
|
||||
},
|
||||
asType: "generation",
|
||||
});
|
||||
|
||||
state.agentState.activeGenerations.set(key, {
|
||||
observation: generation,
|
||||
requestKey: key,
|
||||
ended: false,
|
||||
metadata: captured.metadata ?? {},
|
||||
modelParameters,
|
||||
});
|
||||
state.agentState.generationOrder.push(key);
|
||||
} catch (e) {
|
||||
console.warn("📊 Langfuse: Failed to start generation", e);
|
||||
}
|
||||
}
|
||||
|
||||
export function updateGenerationMetadata(event: Record<string, unknown>) {
|
||||
if (state.isTracingDisabled || !state.agentState) {
|
||||
return;
|
||||
}
|
||||
|
||||
const key = getRequestKey(event, "");
|
||||
const metadata = applyCapturePolicy({ metadata: extractResponseMetadata(event) }, getCapturePolicy()).metadata ?? {};
|
||||
if (!key) {
|
||||
const generation = getOpenGeneration();
|
||||
if (generation) {
|
||||
generation.metadata = { ...generation.metadata, ...metadata };
|
||||
|
||||
const isError =
|
||||
(typeof metadata.status === "number" && metadata.status >= 400) ||
|
||||
event.error ||
|
||||
event.isError;
|
||||
|
||||
if (isError) {
|
||||
generation.observation.update({
|
||||
metadata: generation.metadata,
|
||||
level: "ERROR",
|
||||
statusMessage: String(event.error ?? metadata.statusMessage ?? "Provider request failed")
|
||||
}).end();
|
||||
generation.ended = true;
|
||||
} else {
|
||||
generation.observation.update({ metadata: generation.metadata });
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const generation = state.agentState.activeGenerations.get(key) ?? getOpenGeneration();
|
||||
if (generation) {
|
||||
generation.metadata = { ...generation.metadata, ...metadata };
|
||||
|
||||
const isError =
|
||||
(typeof metadata.status === "number" && metadata.status >= 400) ||
|
||||
event.error ||
|
||||
event.isError;
|
||||
|
||||
if (isError) {
|
||||
generation.observation.update({
|
||||
metadata: generation.metadata,
|
||||
level: "ERROR",
|
||||
statusMessage: String(event.error ?? metadata.statusMessage ?? "Provider request failed")
|
||||
}).end();
|
||||
generation.ended = true;
|
||||
} else {
|
||||
generation.observation.update({ metadata: generation.metadata });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function recordTTFT(event: Record<string, unknown>) {
|
||||
if (state.isTracingDisabled || !state.agentState) {
|
||||
return;
|
||||
}
|
||||
|
||||
const key = getRequestKey(event, "");
|
||||
const generation = key ? state.agentState.activeGenerations.get(key) : getOpenGeneration();
|
||||
|
||||
if (generation && !generation.ttftRecorded && !generation.ended) {
|
||||
generation.ttftRecorded = true;
|
||||
try {
|
||||
generation.observation.update({ completionStartTime: new Date() });
|
||||
} catch (e) {
|
||||
// Ignore
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export async function finishGenerationFromMessage(event: Record<string, unknown>) {
|
||||
if (state.isTracingDisabled || !state.agentState) {
|
||||
return;
|
||||
}
|
||||
|
||||
const message = getMessageFromEvent(event);
|
||||
if (!message || message.role !== "assistant") {
|
||||
return;
|
||||
}
|
||||
|
||||
const generation = getOpenGeneration();
|
||||
const rawOutput = extractAssistantOutput(message);
|
||||
const captured = applyCapturePolicy({ output: rawOutput }, getCapturePolicy());
|
||||
const output = captured.output;
|
||||
state.agentState.latestAssistantOutput = output;
|
||||
|
||||
if (!generation) {
|
||||
return;
|
||||
}
|
||||
|
||||
const usageDetails = extractUsage({ ...event, message });
|
||||
const costDetails = extractCostDetails({ ...event, message });
|
||||
const modelParameters = extractModelParameters(getProviderPayload(event)) ?? generation.modelParameters;
|
||||
const model = String(message.model ?? event.model ?? state.currentModel ?? "");
|
||||
const update: ObservationUpdate = {
|
||||
output,
|
||||
model: model || undefined,
|
||||
modelParameters,
|
||||
usageDetails,
|
||||
...(costDetails ? { costDetails } : {}),
|
||||
metadata: {
|
||||
...generation.metadata,
|
||||
finishReason: message.finishReason ?? message.stopReason ?? event.finishReason,
|
||||
},
|
||||
};
|
||||
update.metadata = applyCapturePolicy({ metadata: update.metadata }, getCapturePolicy()).metadata;
|
||||
|
||||
try {
|
||||
generation.observation.update(update).end();
|
||||
generation.ended = true;
|
||||
} catch (e) {
|
||||
console.warn("📊 Langfuse: Failed to finish generation", e);
|
||||
}
|
||||
}
|
||||
|
||||
export async function createFallbackGenerationFromTurn(event: Record<string, unknown>, message: Record<string, unknown>) {
|
||||
if (state.isTracingDisabled || !state.agentState?.root || state.agentState.generationOrder.length > 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const usageDetails = extractUsage({ ...event, message });
|
||||
const costDetails = extractCostDetails({ ...event, message });
|
||||
const modelParameters = extractModelParameters(getProviderPayload(event));
|
||||
const model = String(message.model ?? event.model ?? state.currentModel ?? "");
|
||||
const captured = applyCapturePolicy(
|
||||
{
|
||||
input: state.agentState.promptInput,
|
||||
output: extractAssistantOutput(message),
|
||||
metadata: {
|
||||
provider: state.currentProvider || undefined,
|
||||
sourceEvent: "turn_end",
|
||||
},
|
||||
},
|
||||
getCapturePolicy(),
|
||||
);
|
||||
const parent = state.agentState.activeTurn ?? state.agentState.root;
|
||||
const generation = await startChildObservation({
|
||||
parent,
|
||||
runtime: getRuntime,
|
||||
name: "llm-generation",
|
||||
body: {
|
||||
input: captured.input,
|
||||
output: captured.output,
|
||||
model: model || undefined,
|
||||
modelParameters,
|
||||
usageDetails,
|
||||
...(costDetails ? { costDetails } : {}),
|
||||
metadata: captured.metadata,
|
||||
},
|
||||
asType: "generation",
|
||||
});
|
||||
|
||||
generation.end();
|
||||
state.agentState.generationOrder.push("turn-end-fallback");
|
||||
} catch (e) {
|
||||
console.warn("📊 Langfuse: Failed to create fallback generation", e);
|
||||
}
|
||||
}
|
||||
156
extensions/pi-langfuse/src/handlers/tool.ts
Normal file
156
extensions/pi-langfuse/src/handlers/tool.ts
Normal file
@@ -0,0 +1,156 @@
|
||||
import { state } from "../state.js";
|
||||
import { getRuntime, sendScore } from "../langfuse.js";
|
||||
import { startChildObservation } from "../observation.js";
|
||||
import {
|
||||
getToolCallId,
|
||||
getToolName,
|
||||
getToolInput,
|
||||
shapePayload,
|
||||
extractTextContent,
|
||||
truncate,
|
||||
estimatePayloadBytes,
|
||||
getCapturePolicy,
|
||||
getLimits,
|
||||
} from "../utils.js";
|
||||
import { applyCapturePolicy } from "../capture-policy.js";
|
||||
import { redactString } from "../redaction.js";
|
||||
|
||||
export async function startToolObservation(event: Record<string, unknown>) {
|
||||
if (state.isTracingDisabled || !state.agentState?.root) {
|
||||
return;
|
||||
}
|
||||
|
||||
const toolCallId = getToolCallId(event);
|
||||
if (!toolCallId || state.agentState.activeTools.has(toolCallId)) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const toolName = getToolName(event);
|
||||
const toolInput = getToolInput(event);
|
||||
const shapedInput = shapePayload(toolInput, { maxString: getLimits().maxToolPayload });
|
||||
const captured = applyCapturePolicy(
|
||||
{
|
||||
toolInput: shapedInput,
|
||||
metadata: { toolName, toolCallId },
|
||||
},
|
||||
getCapturePolicy(),
|
||||
);
|
||||
const inputBytes = estimatePayloadBytes(captured.toolInput, getLimits().maxToolPayload);
|
||||
const parent = state.agentState.activeTurn ?? state.agentState.root;
|
||||
const tool = await startChildObservation({
|
||||
parent,
|
||||
runtime: getRuntime,
|
||||
name: toolName,
|
||||
body: {
|
||||
input: captured.toolInput,
|
||||
metadata: { ...(captured.metadata ?? {}), inputBytes },
|
||||
},
|
||||
asType: "tool",
|
||||
});
|
||||
|
||||
state.toolCallCount++;
|
||||
state.agentState.activeTools.set(toolCallId, {
|
||||
observation: tool,
|
||||
toolName,
|
||||
ended: false,
|
||||
startedAt: Date.now(),
|
||||
inputBytes,
|
||||
});
|
||||
} catch (e) {
|
||||
console.warn("📊 Langfuse: Failed to start tool observation", e);
|
||||
}
|
||||
}
|
||||
|
||||
export async function finishToolObservation(event: Record<string, unknown>) {
|
||||
if (state.isTracingDisabled || !state.agentState) {
|
||||
return;
|
||||
}
|
||||
|
||||
const toolCallId = getToolCallId(event);
|
||||
if (!toolCallId) {
|
||||
return;
|
||||
}
|
||||
|
||||
const activeTool = state.agentState.activeTools.get(toolCallId);
|
||||
if (!activeTool || activeTool.ended) {
|
||||
return;
|
||||
}
|
||||
|
||||
const isError = Boolean(event.isError ?? event.error ?? event.status === "error");
|
||||
const output =
|
||||
extractTextContent(event.content, getLimits().maxToolPayload) ??
|
||||
event.output ??
|
||||
event.result ??
|
||||
event.error ??
|
||||
event.content ??
|
||||
event;
|
||||
|
||||
try {
|
||||
const shapedOutput = shapePayload(output, { maxString: getLimits().maxToolPayload });
|
||||
const captured = applyCapturePolicy(
|
||||
{
|
||||
toolOutput: shapedOutput,
|
||||
metadata: {
|
||||
toolName: activeTool.toolName,
|
||||
toolCallId,
|
||||
isError,
|
||||
},
|
||||
},
|
||||
getCapturePolicy(),
|
||||
);
|
||||
const outputBytes = estimatePayloadBytes(captured.toolOutput, getLimits().maxToolPayload);
|
||||
const durationMs = Math.max(0, Date.now() - activeTool.startedAt);
|
||||
|
||||
activeTool.observation
|
||||
.update({
|
||||
output: captured.toolOutput,
|
||||
level: isError ? "ERROR" : "DEFAULT",
|
||||
statusMessage: isError ? redactString(truncate(String(event.error ?? output), 1_000)) : undefined,
|
||||
metadata: {
|
||||
...(captured.metadata ?? {}),
|
||||
durationMs,
|
||||
inputBytes: activeTool.inputBytes,
|
||||
outputBytes,
|
||||
},
|
||||
})
|
||||
.end();
|
||||
activeTool.ended = true;
|
||||
|
||||
if (isError) {
|
||||
state.errorCount++;
|
||||
await sendScore("tool_is_error", 1, {
|
||||
traceId: state.agentState.traceId,
|
||||
observationId: activeTool.observation.id,
|
||||
});
|
||||
}
|
||||
} catch (e) {
|
||||
console.warn("📊 Langfuse: Failed to finish tool observation", e);
|
||||
} finally {
|
||||
state.agentState.activeTools.delete(toolCallId);
|
||||
}
|
||||
}
|
||||
|
||||
export function closeDanglingObservations(statusMessage: string) {
|
||||
if (state.isTracingDisabled || !state.agentState) {
|
||||
return;
|
||||
}
|
||||
|
||||
for (const activeTool of state.agentState.activeTools.values()) {
|
||||
if (!activeTool.ended) {
|
||||
activeTool.observation
|
||||
.update({ level: "WARNING", statusMessage, metadata: { toolName: activeTool.toolName, cancelled: true } })
|
||||
.end();
|
||||
activeTool.ended = true;
|
||||
}
|
||||
}
|
||||
|
||||
for (const generation of state.agentState.activeGenerations.values()) {
|
||||
if (!generation.ended) {
|
||||
generation.observation.update({ level: "WARNING", statusMessage, metadata: { ...generation.metadata, cancelled: true } }).end();
|
||||
generation.ended = true;
|
||||
}
|
||||
}
|
||||
|
||||
state.agentState.activeTools.clear();
|
||||
}
|
||||
55
extensions/pi-langfuse/src/handlers/turn.ts
Normal file
55
extensions/pi-langfuse/src/handlers/turn.ts
Normal file
@@ -0,0 +1,55 @@
|
||||
import { state } from "../state.js";
|
||||
import { getRuntime } from "../langfuse.js";
|
||||
import { startChildObservation } from "../observation.js";
|
||||
import { shapePayload, getCapturePolicy } from "../utils.js";
|
||||
import { applyCapturePolicy } from "../capture-policy.js";
|
||||
|
||||
export async function startTurnObservation(event: Record<string, unknown>) {
|
||||
if (state.isTracingDisabled || !state.agentState?.root) {
|
||||
return;
|
||||
}
|
||||
|
||||
// If a turn is already active, close it (fallback safety)
|
||||
if (state.agentState.activeTurn) {
|
||||
state.agentState.activeTurn.end();
|
||||
state.agentState.activeTurn = undefined;
|
||||
}
|
||||
|
||||
try {
|
||||
const turnIndex = event.turnIndex ?? state.turnCount;
|
||||
const captured = applyCapturePolicy(
|
||||
{
|
||||
input: shapePayload(event.context ?? event),
|
||||
metadata: { turnIndex },
|
||||
},
|
||||
getCapturePolicy(),
|
||||
);
|
||||
const observation = await startChildObservation({
|
||||
parent: state.agentState.root,
|
||||
runtime: getRuntime,
|
||||
name: "turn",
|
||||
body: {
|
||||
input: captured.input,
|
||||
metadata: captured.metadata,
|
||||
},
|
||||
asType: "span",
|
||||
});
|
||||
|
||||
state.agentState.activeTurn = observation;
|
||||
} catch (e) {
|
||||
console.warn("📊 Langfuse: Failed to start turn observation", e);
|
||||
}
|
||||
}
|
||||
|
||||
export function finishTurnObservation(_event?: Record<string, unknown>) {
|
||||
if (state.isTracingDisabled || !state.agentState?.activeTurn) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
state.agentState.activeTurn.end();
|
||||
state.agentState.activeTurn = undefined;
|
||||
} catch (e) {
|
||||
console.warn("📊 Langfuse: Failed to finish turn observation", e);
|
||||
}
|
||||
}
|
||||
756
extensions/pi-langfuse/src/langfuse.ts
Normal file
756
extensions/pi-langfuse/src/langfuse.ts
Normal file
@@ -0,0 +1,756 @@
|
||||
import type { LangfuseRuntime, LangfuseScoreClient, PendingScore } from "./types.js";
|
||||
import { state } from "./state.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
|
||||
let runtime: LangfuseRuntime | null = null;
|
||||
let registeredContextManager: OtelContextManager | null = null;
|
||||
const activeSessions = new Set<string>();
|
||||
let lastRuntimeError: { scope: string; message: string; timestamp: Date } | null = null;
|
||||
|
||||
type FallbackObservationType = "SPAN" | "GENERATION";
|
||||
|
||||
interface OtelContextManager {
|
||||
enable(): OtelContextManager;
|
||||
disable(): void;
|
||||
}
|
||||
|
||||
interface OtelContextApi {
|
||||
setGlobalContextManager(contextManager: OtelContextManager): boolean;
|
||||
}
|
||||
|
||||
type AsyncHooksContextManagerCtor = new () => OtelContextManager;
|
||||
|
||||
interface RestFallbackTrace {
|
||||
id: string;
|
||||
timestamp: string;
|
||||
name: string;
|
||||
input?: unknown;
|
||||
output?: unknown;
|
||||
sessionId?: string;
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
interface RestFallbackObservation {
|
||||
id: string;
|
||||
traceId: string;
|
||||
type: FallbackObservationType;
|
||||
name: string;
|
||||
startTime: string;
|
||||
endTime?: string;
|
||||
parentObservationId?: string;
|
||||
input?: unknown;
|
||||
output?: unknown;
|
||||
metadata?: Record<string, unknown>;
|
||||
model?: string;
|
||||
modelParameters?: Record<string, string | number>;
|
||||
usageDetails?: Record<string, number>;
|
||||
costDetails?: Record<string, number>;
|
||||
level?: "DEBUG" | "DEFAULT" | "WARNING" | "ERROR";
|
||||
statusMessage?: string;
|
||||
completionStartTime?: string;
|
||||
}
|
||||
|
||||
interface RestFallbackStore {
|
||||
trace?: RestFallbackTrace;
|
||||
observations: RestFallbackObservation[];
|
||||
observationById: Map<string, RestFallbackObservation>;
|
||||
attempted: boolean;
|
||||
}
|
||||
|
||||
const OTEL_VISIBILITY_TIMEOUT_MS = 1_500;
|
||||
const OTEL_VISIBILITY_POLL_INTERVAL_MS = 200;
|
||||
const DEFAULT_SHUTDOWN_STEP_TIMEOUT_MS = 2_000;
|
||||
const DEFAULT_LANGFUSE_REQUEST_TIMEOUT_SECONDS = 5;
|
||||
const DEFAULT_SCORE_FLUSH_AT = 10;
|
||||
const DEFAULT_SCORE_FLUSH_INTERVAL_MS = 1_000;
|
||||
const MAX_SCORE_QUEUE_SIZE = 100_000;
|
||||
const MAX_SCORE_BATCH_SIZE = 100;
|
||||
|
||||
let shutdownStepTimeoutMs = DEFAULT_SHUTDOWN_STEP_TIMEOUT_MS;
|
||||
|
||||
function nowIso() {
|
||||
return new Date().toISOString();
|
||||
}
|
||||
|
||||
function resolvePositiveEnvNumber(name: string, fallback: number, integer = false): number {
|
||||
const parsed = Number(process.env[name]);
|
||||
if (!Number.isFinite(parsed) || parsed <= 0) {
|
||||
return fallback;
|
||||
}
|
||||
return integer ? Math.floor(parsed) : parsed;
|
||||
}
|
||||
|
||||
function delay(ms: number, signal?: AbortSignal) {
|
||||
return new Promise<void>((resolve, reject) => {
|
||||
if (signal?.aborted) {
|
||||
reject(signal.reason);
|
||||
return;
|
||||
}
|
||||
|
||||
const timeout = setTimeout(resolve, ms);
|
||||
signal?.addEventListener("abort", () => {
|
||||
clearTimeout(timeout);
|
||||
reject(signal.reason);
|
||||
}, { once: true });
|
||||
});
|
||||
}
|
||||
|
||||
function debugLog(message: string) {
|
||||
if (process.env.PI_LANGFUSE_DEBUG === "1" || process.env.PI_LANGFUSE_DEBUG === "true") {
|
||||
console.log(message);
|
||||
}
|
||||
}
|
||||
|
||||
export function ensureOtelContextManager(
|
||||
contextApi: OtelContextApi,
|
||||
AsyncHooksContextManager: AsyncHooksContextManagerCtor,
|
||||
): boolean {
|
||||
if (registeredContextManager) {
|
||||
return true;
|
||||
}
|
||||
|
||||
const contextManager = new AsyncHooksContextManager().enable();
|
||||
if (contextApi.setGlobalContextManager(contextManager)) {
|
||||
registeredContextManager = contextManager;
|
||||
return true;
|
||||
}
|
||||
|
||||
contextManager.disable();
|
||||
return false;
|
||||
}
|
||||
|
||||
function rememberRuntimeError(scope: string, error: unknown) {
|
||||
lastRuntimeError = {
|
||||
scope,
|
||||
message: error instanceof Error ? error.message : String(error),
|
||||
timestamp: new Date(),
|
||||
};
|
||||
}
|
||||
|
||||
export function getLastRuntimeError(): { scope: string; message: string; timestamp: Date } | null {
|
||||
return lastRuntimeError;
|
||||
}
|
||||
|
||||
async function withShutdownDeadline<T>(label: string, startOperation: () => Promise<T> | undefined, deadline: number): Promise<T | undefined> {
|
||||
const remainingMs = deadline - Date.now();
|
||||
if (remainingMs <= 0) {
|
||||
debugLog(`📊 Langfuse: Skipped ${label}; shutdown deadline elapsed`);
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const operation = startOperation();
|
||||
if (!operation) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
let timeout: NodeJS.Timeout | undefined;
|
||||
try {
|
||||
return await Promise.race([
|
||||
operation,
|
||||
new Promise<undefined>((resolve) => {
|
||||
timeout = setTimeout(() => {
|
||||
debugLog(`📊 Langfuse: ${label} timed out; shutdown deadline elapsed`);
|
||||
resolve(undefined);
|
||||
}, remainingMs);
|
||||
}),
|
||||
]);
|
||||
} finally {
|
||||
if (timeout) {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function getRuntimeConfig(rt: LangfuseRuntime) {
|
||||
return rt.runtimeConfig ?? state.config;
|
||||
}
|
||||
|
||||
function ingestionHeaders(rt: LangfuseRuntime): Record<string, string> {
|
||||
const config = getRuntimeConfig(rt);
|
||||
if (!config) {
|
||||
throw new Error("Langfuse runtime config is unavailable");
|
||||
}
|
||||
|
||||
const auth = Buffer.from(`${config.publicKey}:${config.secretKey}`).toString("base64");
|
||||
return {
|
||||
Authorization: `Basic ${auth}`,
|
||||
"Content-Type": "application/json",
|
||||
};
|
||||
}
|
||||
|
||||
async function ingestBatch(rt: LangfuseRuntime, batch: unknown[], signal: AbortSignal): Promise<unknown[]> {
|
||||
const config = getRuntimeConfig(rt);
|
||||
if (!config) {
|
||||
throw new Error("Langfuse runtime config is unavailable");
|
||||
}
|
||||
|
||||
const response = await fetch(`${config.host.replace(/\/$/, "")}/api/public/ingestion`, {
|
||||
method: "POST",
|
||||
headers: ingestionHeaders(rt),
|
||||
body: JSON.stringify({ batch }),
|
||||
signal,
|
||||
});
|
||||
if (!response.ok) {
|
||||
throw new Error(`Langfuse ingestion failed with HTTP ${response.status}`);
|
||||
}
|
||||
|
||||
const text = await response.text();
|
||||
if (!text) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const responseBody = JSON.parse(text) as { errors?: unknown[] };
|
||||
return Array.isArray(responseBody.errors) ? responseBody.errors : [];
|
||||
}
|
||||
|
||||
async function flushPendingScores(rt: LangfuseRuntime, signal: AbortSignal): Promise<void> {
|
||||
const pendingScores = rt.pendingScores;
|
||||
if (!pendingScores || pendingScores.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
while (pendingScores.length > 0) {
|
||||
const scores = pendingScores.slice(0, MAX_SCORE_BATCH_SIZE);
|
||||
try {
|
||||
const errors = await ingestBatch(
|
||||
rt,
|
||||
scores.map((score) => ({
|
||||
type: "score-create",
|
||||
id: randomUUID(),
|
||||
timestamp: nowIso(),
|
||||
body: score,
|
||||
})),
|
||||
signal,
|
||||
);
|
||||
pendingScores.splice(0, scores.length);
|
||||
if (errors.length > 0) {
|
||||
rememberRuntimeError("score ingestion", new Error(JSON.stringify(errors)));
|
||||
console.warn("📊 Langfuse: Score ingestion reported errors", errors);
|
||||
}
|
||||
} catch (error) {
|
||||
if ((error as { name?: string }).name !== "AbortError") {
|
||||
rememberRuntimeError("score ingestion", error);
|
||||
console.warn("📊 Langfuse: Failed to flush scores", error);
|
||||
}
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function clearScoreFlushTimer(rt: LangfuseRuntime) {
|
||||
if (rt.scoreFlushTimer) {
|
||||
clearTimeout(rt.scoreFlushTimer);
|
||||
rt.scoreFlushTimer = undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function scheduleScoreFlush(rt: LangfuseRuntime) {
|
||||
if (
|
||||
rt.scoreFlushStopped
|
||||
|| rt.scoreFlushTimer
|
||||
|| rt.scoreFlushPromise
|
||||
|| !rt.pendingScores?.length
|
||||
) {
|
||||
return;
|
||||
}
|
||||
|
||||
rt.scoreFlushTimer = setTimeout(() => {
|
||||
rt.scoreFlushTimer = undefined;
|
||||
void startScoreFlush(rt);
|
||||
}, rt.scoreFlushIntervalMs ?? DEFAULT_SCORE_FLUSH_INTERVAL_MS);
|
||||
rt.scoreFlushTimer.unref?.();
|
||||
}
|
||||
|
||||
function startScoreFlush(rt: LangfuseRuntime): Promise<void> {
|
||||
if (rt.scoreFlushPromise) {
|
||||
return rt.scoreFlushPromise;
|
||||
}
|
||||
|
||||
clearScoreFlushTimer(rt);
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(
|
||||
() => controller.abort(new DOMException("Langfuse score request timed out", "AbortError")),
|
||||
rt.scoreRequestTimeoutMs ?? DEFAULT_LANGFUSE_REQUEST_TIMEOUT_SECONDS * 1_000,
|
||||
);
|
||||
timeout.unref?.();
|
||||
rt.scoreFlushController = controller;
|
||||
|
||||
const promise = flushPendingScores(rt, controller.signal).finally(() => {
|
||||
clearTimeout(timeout);
|
||||
if (rt.scoreFlushPromise === promise) {
|
||||
rt.scoreFlushPromise = undefined;
|
||||
}
|
||||
if (rt.scoreFlushController === controller) {
|
||||
rt.scoreFlushController = undefined;
|
||||
}
|
||||
scheduleScoreFlush(rt);
|
||||
});
|
||||
rt.scoreFlushPromise = promise;
|
||||
return promise;
|
||||
}
|
||||
|
||||
function stopScoreFlush(rt: LangfuseRuntime) {
|
||||
rt.scoreFlushStopped = true;
|
||||
clearScoreFlushTimer(rt);
|
||||
rt.scoreFlushController?.abort(
|
||||
new DOMException("Langfuse score flushing stopped", "AbortError"),
|
||||
);
|
||||
}
|
||||
|
||||
function toIso(value: unknown): string | undefined {
|
||||
if (!value) {
|
||||
return undefined;
|
||||
}
|
||||
if (value instanceof Date) {
|
||||
return value.toISOString();
|
||||
}
|
||||
if (typeof value === "string") {
|
||||
return value;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function mergeMetadata(current: Record<string, unknown> | undefined, next: Record<string, unknown> | undefined) {
|
||||
return next ? { ...(current ?? {}), ...next } : current;
|
||||
}
|
||||
|
||||
function applyObservationUpdate(record: RestFallbackObservation, body: Record<string, unknown> | undefined) {
|
||||
if (!body) {
|
||||
return;
|
||||
}
|
||||
|
||||
if ("input" in body) record.input = body.input;
|
||||
if ("output" in body) record.output = body.output;
|
||||
if ("metadata" in body && body.metadata && typeof body.metadata === "object") {
|
||||
record.metadata = mergeMetadata(record.metadata, body.metadata as Record<string, unknown>);
|
||||
}
|
||||
if (typeof body.model === "string") record.model = body.model;
|
||||
if (body.modelParameters && typeof body.modelParameters === "object") {
|
||||
record.modelParameters = body.modelParameters as Record<string, string | number>;
|
||||
}
|
||||
if (body.usageDetails && typeof body.usageDetails === "object") {
|
||||
record.usageDetails = body.usageDetails as Record<string, number>;
|
||||
}
|
||||
if (body.costDetails && typeof body.costDetails === "object") {
|
||||
record.costDetails = body.costDetails as Record<string, number>;
|
||||
}
|
||||
if (typeof body.level === "string") record.level = body.level as RestFallbackObservation["level"];
|
||||
if (typeof body.statusMessage === "string") record.statusMessage = body.statusMessage;
|
||||
const completionStartTime = toIso(body.completionStartTime);
|
||||
if (completionStartTime) record.completionStartTime = completionStartTime;
|
||||
}
|
||||
|
||||
function applyTraceUpdate(store: RestFallbackStore, body: Record<string, unknown> | undefined) {
|
||||
if (!store.trace || !body) {
|
||||
return;
|
||||
}
|
||||
|
||||
if ("input" in body) store.trace.input = body.input;
|
||||
if ("output" in body) store.trace.output = body.output;
|
||||
if ("metadata" in body && body.metadata && typeof body.metadata === "object") {
|
||||
store.trace.metadata = mergeMetadata(store.trace.metadata, body.metadata as Record<string, unknown>);
|
||||
}
|
||||
}
|
||||
|
||||
function observationType(asType?: string): FallbackObservationType {
|
||||
return asType === "generation" ? "GENERATION" : "SPAN";
|
||||
}
|
||||
|
||||
function wrapObservation(
|
||||
observation: any,
|
||||
store: RestFallbackStore,
|
||||
name: string,
|
||||
body: Record<string, unknown> | undefined,
|
||||
asType?: string,
|
||||
parentObservationId?: string,
|
||||
): any {
|
||||
const id = observation.id || randomUUID();
|
||||
const traceId = observation.traceId || store.trace?.id || randomUUID();
|
||||
const metadata = body?.metadata && typeof body.metadata === "object" ? body.metadata as Record<string, unknown> : undefined;
|
||||
const record: RestFallbackObservation = {
|
||||
id,
|
||||
traceId,
|
||||
name,
|
||||
type: observationType(asType),
|
||||
startTime: nowIso(),
|
||||
parentObservationId,
|
||||
metadata: mergeMetadata(metadata, asType && asType !== "generation" && asType !== "span" ? { langfuseObservationType: asType } : undefined),
|
||||
};
|
||||
applyObservationUpdate(record, body);
|
||||
|
||||
store.observations.push(record);
|
||||
store.observationById.set(id, record);
|
||||
|
||||
if (!parentObservationId && !store.trace) {
|
||||
store.trace = {
|
||||
id: traceId,
|
||||
timestamp: record.startTime,
|
||||
name,
|
||||
input: body?.input,
|
||||
sessionId: typeof metadata?.sessionId === "string" ? metadata.sessionId : state.currentSessionId || undefined,
|
||||
metadata,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
...observation,
|
||||
id,
|
||||
traceId,
|
||||
update(updateBody?: Record<string, unknown>) {
|
||||
applyObservationUpdate(record, updateBody);
|
||||
if (!parentObservationId) {
|
||||
applyTraceUpdate(store, updateBody);
|
||||
}
|
||||
const updated = observation.update(updateBody);
|
||||
return updated === observation ? this : updated;
|
||||
},
|
||||
end(endBody?: Record<string, unknown>) {
|
||||
if (endBody && typeof endBody === "object") {
|
||||
applyObservationUpdate(record, endBody);
|
||||
if (!parentObservationId) {
|
||||
applyTraceUpdate(store, endBody);
|
||||
}
|
||||
}
|
||||
record.endTime = nowIso();
|
||||
return observation.end();
|
||||
},
|
||||
startObservation(childName: string, childBody?: Record<string, unknown>, options?: { asType?: string }) {
|
||||
const child = observation.startObservation(childName, childBody, options);
|
||||
return wrapObservation(child, store, childName, childBody, options?.asType, id);
|
||||
},
|
||||
setTraceIO(traceBody?: { input?: unknown; output?: unknown }) {
|
||||
applyTraceUpdate(store, traceBody);
|
||||
return observation.setTraceIO?.(traceBody);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function traceExists(rt: LangfuseRuntime, traceId: string, signal: AbortSignal): Promise<boolean> {
|
||||
const config = getRuntimeConfig(rt);
|
||||
if (!config) {
|
||||
return false;
|
||||
}
|
||||
|
||||
try {
|
||||
const response = await fetch(
|
||||
`${config.host.replace(/\/$/, "")}/api/public/traces/${encodeURIComponent(traceId)}`,
|
||||
{
|
||||
headers: ingestionHeaders(rt),
|
||||
signal,
|
||||
},
|
||||
);
|
||||
if (response.status === 404) {
|
||||
return false;
|
||||
}
|
||||
if (!response.ok) {
|
||||
throw new Error(`Langfuse trace visibility check failed with HTTP ${response.status}`);
|
||||
}
|
||||
return true;
|
||||
} catch (error) {
|
||||
if (signal.aborted) {
|
||||
throw error;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function waitForTraceVisibility(rt: LangfuseRuntime, traceId: string, signal: AbortSignal): Promise<boolean> {
|
||||
const deadline = Date.now() + OTEL_VISIBILITY_TIMEOUT_MS;
|
||||
|
||||
while (true) {
|
||||
if (await traceExists(rt, traceId, signal)) {
|
||||
return true;
|
||||
}
|
||||
|
||||
const remainingMs = deadline - Date.now();
|
||||
if (remainingMs <= 0) {
|
||||
return false;
|
||||
}
|
||||
|
||||
await delay(Math.min(OTEL_VISIBILITY_POLL_INTERVAL_MS, remainingMs), signal);
|
||||
}
|
||||
}
|
||||
|
||||
function eventTimestamp(record: { endTime?: string; startTime?: string; timestamp?: string }) {
|
||||
return record.endTime ?? record.startTime ?? record.timestamp ?? nowIso();
|
||||
}
|
||||
|
||||
async function fallbackToRestIngestion(rt: LangfuseRuntime, signal: AbortSignal) {
|
||||
const store = rt.restFallback as RestFallbackStore | undefined;
|
||||
if (!store?.trace || store.attempted) {
|
||||
return;
|
||||
}
|
||||
store.attempted = true;
|
||||
|
||||
if (await waitForTraceVisibility(rt, store.trace.id, signal)) {
|
||||
return;
|
||||
}
|
||||
|
||||
const trace = store.trace;
|
||||
const batch: any[] = [
|
||||
{
|
||||
type: "trace-create",
|
||||
id: randomUUID(),
|
||||
timestamp: eventTimestamp(trace),
|
||||
body: {
|
||||
id: trace.id,
|
||||
timestamp: trace.timestamp,
|
||||
name: trace.name,
|
||||
input: trace.input,
|
||||
output: trace.output,
|
||||
sessionId: trace.sessionId,
|
||||
metadata: trace.metadata,
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
for (const observation of store.observations) {
|
||||
const body = {
|
||||
id: observation.id,
|
||||
traceId: observation.traceId,
|
||||
name: observation.name,
|
||||
startTime: observation.startTime,
|
||||
endTime: observation.endTime,
|
||||
input: observation.input,
|
||||
output: observation.output,
|
||||
metadata: observation.metadata,
|
||||
level: observation.level,
|
||||
statusMessage: observation.statusMessage,
|
||||
parentObservationId: observation.parentObservationId,
|
||||
...(observation.type === "GENERATION"
|
||||
? {
|
||||
completionStartTime: observation.completionStartTime,
|
||||
model: observation.model,
|
||||
modelParameters: observation.modelParameters,
|
||||
usageDetails: observation.usageDetails,
|
||||
costDetails: observation.costDetails,
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
batch.push({
|
||||
type: observation.type === "GENERATION" ? "generation-create" : "span-create",
|
||||
id: randomUUID(),
|
||||
timestamp: eventTimestamp(observation),
|
||||
body,
|
||||
});
|
||||
}
|
||||
|
||||
const errors = await ingestBatch(rt, batch, signal);
|
||||
if (errors.length > 0) {
|
||||
rememberRuntimeError("REST fallback ingestion", new Error(JSON.stringify(errors)));
|
||||
console.warn("📊 Langfuse: REST fallback ingestion reported errors", errors);
|
||||
} else {
|
||||
debugLog(`📊 Langfuse: OTel trace ${trace.id} was not visible; wrote fallback trace via REST ingestion`);
|
||||
}
|
||||
}
|
||||
|
||||
export async function getRuntime(): Promise<LangfuseRuntime> {
|
||||
if (!state.config) {
|
||||
throw new Error("Langfuse config is not set");
|
||||
}
|
||||
|
||||
// Track the current session as a runtime consumer.
|
||||
// Multiple sessions can share the same runtime; shutdown is deferred
|
||||
// until the last session releases it.
|
||||
const sessionId = state.currentSessionId;
|
||||
if (sessionId) {
|
||||
activeSessions.add(sessionId);
|
||||
}
|
||||
|
||||
if (!runtime) {
|
||||
const [
|
||||
{ BasicTracerProvider },
|
||||
{ context },
|
||||
{ AsyncHooksContextManager },
|
||||
{ LangfuseSpanProcessor },
|
||||
tracing,
|
||||
{ LangfuseClient },
|
||||
] = await Promise.all([
|
||||
import("@opentelemetry/sdk-trace-base"),
|
||||
import("@opentelemetry/api"),
|
||||
import("@opentelemetry/context-async-hooks"),
|
||||
import("@langfuse/otel"),
|
||||
import("@langfuse/tracing"),
|
||||
import("@langfuse/client"),
|
||||
]);
|
||||
|
||||
const restFallback: RestFallbackStore = {
|
||||
observations: [],
|
||||
observationById: new Map(),
|
||||
attempted: false,
|
||||
};
|
||||
|
||||
try {
|
||||
ensureOtelContextManager(context, AsyncHooksContextManager);
|
||||
const scoreFlushAt = resolvePositiveEnvNumber("LANGFUSE_FLUSH_AT", DEFAULT_SCORE_FLUSH_AT, true);
|
||||
const scoreFlushIntervalMs =
|
||||
resolvePositiveEnvNumber("LANGFUSE_FLUSH_INTERVAL", DEFAULT_SCORE_FLUSH_INTERVAL_MS / 1_000) * 1_000;
|
||||
const scoreRequestTimeoutMs =
|
||||
resolvePositiveEnvNumber("LANGFUSE_TIMEOUT", DEFAULT_LANGFUSE_REQUEST_TIMEOUT_SECONDS) * 1_000;
|
||||
const spanProcessor = new LangfuseSpanProcessor({
|
||||
publicKey: state.config.publicKey,
|
||||
secretKey: state.config.secretKey,
|
||||
baseUrl: state.config.host,
|
||||
});
|
||||
const tracerProvider = new BasicTracerProvider({ spanProcessors: [spanProcessor] });
|
||||
tracing.setLangfuseTracerProvider(tracerProvider);
|
||||
|
||||
runtime = {
|
||||
startObservation: ((name: string, body?: Record<string, unknown>, options?: { asType?: string }) => {
|
||||
const observation = (tracing as any).startObservation(name, body, options);
|
||||
return wrapObservation(observation, restFallback, name, body, options?.asType);
|
||||
}) as unknown as LangfuseRuntime["startObservation"],
|
||||
propagateAttributes: tracing.propagateAttributes as unknown as LangfuseRuntime["propagateAttributes"],
|
||||
scoreClient: new LangfuseClient({
|
||||
publicKey: state.config.publicKey,
|
||||
secretKey: state.config.secretKey,
|
||||
baseUrl: state.config.host,
|
||||
}) as LangfuseScoreClient,
|
||||
spanProcessor,
|
||||
tracerProvider,
|
||||
clearTracerProvider: () => tracing.setLangfuseTracerProvider(null),
|
||||
restFallback,
|
||||
pendingScores: [],
|
||||
scoreFlushAt,
|
||||
scoreFlushIntervalMs,
|
||||
scoreRequestTimeoutMs,
|
||||
scoreFlushStopped: false,
|
||||
runtimeConfig: {
|
||||
publicKey: state.config.publicKey,
|
||||
secretKey: state.config.secretKey,
|
||||
host: state.config.host,
|
||||
},
|
||||
};
|
||||
lastRuntimeError = null;
|
||||
} catch (e) {
|
||||
rememberRuntimeError("runtime init", e);
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
return runtime as LangfuseRuntime;
|
||||
}
|
||||
|
||||
function doShutdownRuntime(): Promise<void> {
|
||||
return (async () => {
|
||||
if (!runtime) {
|
||||
return;
|
||||
}
|
||||
|
||||
const rt = runtime;
|
||||
runtime = null;
|
||||
const deadline = Date.now() + shutdownStepTimeoutMs;
|
||||
const controller = new AbortController();
|
||||
const abortTimeout = setTimeout(() => controller.abort(), shutdownStepTimeoutMs);
|
||||
stopScoreFlush(rt);
|
||||
|
||||
try {
|
||||
await withShutdownDeadline(
|
||||
"Active score flush",
|
||||
() => rt.scoreFlushPromise,
|
||||
deadline,
|
||||
);
|
||||
await withShutdownDeadline("OTel force flush", () => rt.tracerProvider?.forceFlush?.(), deadline);
|
||||
await withShutdownDeadline(
|
||||
"REST fallback ingestion",
|
||||
() => fallbackToRestIngestion(rt, controller.signal),
|
||||
deadline,
|
||||
);
|
||||
await flushPendingScores(rt, controller.signal);
|
||||
await withShutdownDeadline("Langfuse score flush", () => rt.scoreClient.flush?.(), deadline);
|
||||
await withShutdownDeadline("Langfuse client shutdown", () => rt.scoreClient.shutdown?.(), deadline);
|
||||
await withShutdownDeadline("OTel tracer shutdown", () => rt.tracerProvider?.shutdown?.(), deadline);
|
||||
} catch (e) {
|
||||
rememberRuntimeError("runtime shutdown", e);
|
||||
console.warn("📊 Langfuse: Failed to flush/shutdown cleanly", e);
|
||||
} finally {
|
||||
clearTimeout(abortTimeout);
|
||||
clearScoreFlushTimer(rt);
|
||||
rt.scoreFlushController?.abort();
|
||||
rt.scoreFlushController = undefined;
|
||||
rt.scoreFlushPromise = undefined;
|
||||
if (!runtime) {
|
||||
rt.clearTracerProvider?.();
|
||||
}
|
||||
}
|
||||
})();
|
||||
}
|
||||
|
||||
/**
|
||||
* Release the current session's reference to the Langfuse runtime.
|
||||
* Only actually shuts down the runtime when the last session releases it.
|
||||
* Accepts an optional sessionId for use outside of withSession (e.g. deferred callbacks).
|
||||
*/
|
||||
export async function shutdownRuntime(sessionId?: string): Promise<void> {
|
||||
const sid = sessionId ?? state.currentSessionId;
|
||||
if (sid) {
|
||||
activeSessions.delete(sid);
|
||||
}
|
||||
|
||||
// Still have active sessions — keep the runtime alive.
|
||||
if (activeSessions.size > 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
await doShutdownRuntime();
|
||||
}
|
||||
|
||||
/**
|
||||
* Force-shutdown the Langfuse runtime regardless of active session references.
|
||||
* Used when the user manually reconfigures (e.g. /langfuse-setup) and needs
|
||||
* a fresh runtime with new credentials.
|
||||
*/
|
||||
export async function forceShutdownRuntime(): Promise<void> {
|
||||
activeSessions.clear();
|
||||
await doShutdownRuntime();
|
||||
}
|
||||
|
||||
export function __setRuntimeForTest(rt: LangfuseRuntime | null, timeoutMs = DEFAULT_SHUTDOWN_STEP_TIMEOUT_MS): void {
|
||||
if (runtime && runtime !== rt) {
|
||||
stopScoreFlush(runtime);
|
||||
}
|
||||
runtime = rt;
|
||||
if (rt) {
|
||||
rt.scoreFlushStopped = false;
|
||||
}
|
||||
shutdownStepTimeoutMs = timeoutMs;
|
||||
activeSessions.clear();
|
||||
}
|
||||
|
||||
export async function sendScore(name: string, value: number, options: { traceId?: string; observationId?: string } = {}) {
|
||||
try {
|
||||
const rt = await getRuntime();
|
||||
const score: PendingScore = {
|
||||
name,
|
||||
value,
|
||||
dataType: name === "session_had_errors" || name === "tool_is_error" ? "BOOLEAN" : "NUMERIC",
|
||||
traceId: options.traceId,
|
||||
observationId: options.observationId,
|
||||
sessionId: options.traceId ? undefined : state.currentSessionId || undefined,
|
||||
...(process.env.LANGFUSE_TRACING_ENVIRONMENT
|
||||
? { environment: process.env.LANGFUSE_TRACING_ENVIRONMENT }
|
||||
: {}),
|
||||
};
|
||||
if (!rt.pendingScores) {
|
||||
return;
|
||||
}
|
||||
if (rt.pendingScores.length >= MAX_SCORE_QUEUE_SIZE) {
|
||||
const error = new Error(
|
||||
`Langfuse score queue is full (${MAX_SCORE_QUEUE_SIZE}); dropping score`,
|
||||
);
|
||||
rememberRuntimeError("score queue", error);
|
||||
console.warn(`📊 Langfuse: ${error.message}`);
|
||||
return;
|
||||
}
|
||||
|
||||
rt.pendingScores.push(score);
|
||||
if (rt.pendingScores.length >= (rt.scoreFlushAt ?? DEFAULT_SCORE_FLUSH_AT)) {
|
||||
void startScoreFlush(rt);
|
||||
} else {
|
||||
scheduleScoreFlush(rt);
|
||||
}
|
||||
} catch (e) {
|
||||
rememberRuntimeError(`score ${name}`, e);
|
||||
console.warn(`📊 Langfuse: Failed to send score ${name}`, e);
|
||||
}
|
||||
}
|
||||
97
extensions/pi-langfuse/src/limits.ts
Normal file
97
extensions/pi-langfuse/src/limits.ts
Normal file
@@ -0,0 +1,97 @@
|
||||
import {
|
||||
MAX_ARRAY_ITEMS,
|
||||
MAX_DEPTH,
|
||||
MAX_OBJECT_KEYS,
|
||||
MAX_PAYLOAD_NODES,
|
||||
MAX_STRING_LENGTH,
|
||||
MAX_TOOL_PAYLOAD_LENGTH,
|
||||
} from "./constants.js";
|
||||
import type { EnvLike } from "./capture-policy.js";
|
||||
import { state } from "./state.js";
|
||||
|
||||
/**
|
||||
* Payload-shaping limits. Every field is a positive integer, or
|
||||
* `Number.POSITIVE_INFINITY` to disable that limit entirely (capture everything).
|
||||
* Resolved once from the environment and stored on the loaded config; consumers
|
||||
* read the resolved values via `getLimits()` rather than the raw constants.
|
||||
*/
|
||||
export interface PayloadLimits {
|
||||
/** Max characters kept per captured string (generation/agent inputs, outputs, system prompt). */
|
||||
readonly maxString: number;
|
||||
/** Max characters kept for tool inputs/outputs (their payloads run larger than chat strings). */
|
||||
readonly maxToolPayload: number;
|
||||
/** Max nesting depth walked when shaping a structured payload. */
|
||||
readonly maxDepth: number;
|
||||
/** Max array elements kept per array. */
|
||||
readonly maxArrayItems: number;
|
||||
/** Max own-keys kept per object. */
|
||||
readonly maxObjectKeys: number;
|
||||
/** Max total nodes visited across a whole payload before bailing with `[payload too large]`. */
|
||||
readonly maxNodes: number;
|
||||
}
|
||||
|
||||
export const DEFAULT_LIMITS: PayloadLimits = {
|
||||
maxString: MAX_STRING_LENGTH,
|
||||
maxToolPayload: MAX_TOOL_PAYLOAD_LENGTH,
|
||||
maxDepth: MAX_DEPTH,
|
||||
maxArrayItems: MAX_ARRAY_ITEMS,
|
||||
maxObjectKeys: MAX_OBJECT_KEYS,
|
||||
maxNodes: MAX_PAYLOAD_NODES,
|
||||
};
|
||||
|
||||
/** Words that mean "no limit" when supplied as an env value. */
|
||||
const UNLIMITED_WORDS = new Set(["off", "none", "false", "no", "unlimited", "inf", "infinity"]);
|
||||
|
||||
/**
|
||||
* Parse one limit env value.
|
||||
* - unset / blank / unparseable -> `fallback` (the built-in default)
|
||||
* - "off"/"none"/"unlimited"/... or a value <= 0 -> `Infinity` (limit removed)
|
||||
* - a positive number -> that integer
|
||||
*/
|
||||
export function parseLimit(raw: string | undefined, fallback: number): number {
|
||||
if (raw === undefined) {
|
||||
return fallback;
|
||||
}
|
||||
const trimmed = raw.trim().toLowerCase();
|
||||
if (trimmed === "") {
|
||||
return fallback;
|
||||
}
|
||||
if (UNLIMITED_WORDS.has(trimmed)) {
|
||||
return Number.POSITIVE_INFINITY;
|
||||
}
|
||||
const parsed = Number(trimmed);
|
||||
if (!Number.isFinite(parsed)) {
|
||||
return fallback;
|
||||
}
|
||||
if (parsed <= 0) {
|
||||
return Number.POSITIVE_INFINITY;
|
||||
}
|
||||
return Math.floor(parsed);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve payload limits from the environment. Each `PI_LANGFUSE_MAX_*` var overrides
|
||||
* the corresponding default; set any to `0`/`off`/`unlimited` to remove that limit.
|
||||
* Namespaced `PI_LANGFUSE_*` (not `LANGFUSE_*`) to avoid clashing with Langfuse
|
||||
* server env vars such as `LANGFUSE_MAX_EVENT_SIZE_BYTES`.
|
||||
*/
|
||||
export function createPayloadLimits(env: EnvLike = process.env as EnvLike): PayloadLimits {
|
||||
return {
|
||||
maxString: parseLimit(env.PI_LANGFUSE_MAX_STRING_LENGTH, DEFAULT_LIMITS.maxString),
|
||||
maxToolPayload: parseLimit(env.PI_LANGFUSE_MAX_TOOL_PAYLOAD_LENGTH, DEFAULT_LIMITS.maxToolPayload),
|
||||
maxDepth: parseLimit(env.PI_LANGFUSE_MAX_DEPTH, DEFAULT_LIMITS.maxDepth),
|
||||
maxArrayItems: parseLimit(env.PI_LANGFUSE_MAX_ARRAY_ITEMS, DEFAULT_LIMITS.maxArrayItems),
|
||||
maxObjectKeys: parseLimit(env.PI_LANGFUSE_MAX_OBJECT_KEYS, DEFAULT_LIMITS.maxObjectKeys),
|
||||
maxNodes: parseLimit(env.PI_LANGFUSE_MAX_PAYLOAD_NODES, DEFAULT_LIMITS.maxNodes),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolved limits for the current session: the config-loaded values when a
|
||||
* config is active, otherwise a fresh resolve from the environment. Every
|
||||
* capture/redaction path reads limits through this so a single env change
|
||||
* (or config) governs truncation everywhere.
|
||||
*/
|
||||
export function getLimits(): PayloadLimits {
|
||||
return state.config?.limits ?? createPayloadLimits();
|
||||
}
|
||||
21
extensions/pi-langfuse/src/observation.ts
Normal file
21
extensions/pi-langfuse/src/observation.ts
Normal file
@@ -0,0 +1,21 @@
|
||||
import type { LangfuseObservation, LangfuseRuntime, ObservationUpdate } from "./types.js";
|
||||
|
||||
export async function startChildObservation({
|
||||
parent,
|
||||
runtime,
|
||||
name,
|
||||
body,
|
||||
asType,
|
||||
}: {
|
||||
parent: LangfuseObservation;
|
||||
runtime: () => Promise<LangfuseRuntime>;
|
||||
name: string;
|
||||
body?: ObservationUpdate;
|
||||
asType: "generation" | "tool" | "span";
|
||||
}): Promise<LangfuseObservation> {
|
||||
if (parent.startObservation) {
|
||||
return parent.startObservation(name, body, { asType });
|
||||
}
|
||||
|
||||
return (await runtime()).startObservation(name, body, { asType });
|
||||
}
|
||||
119
extensions/pi-langfuse/src/redaction.ts
Normal file
119
extensions/pi-langfuse/src/redaction.ts
Normal file
@@ -0,0 +1,119 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { getLimits } from "./limits.js";
|
||||
|
||||
export const REDACTED = "[REDACTED_SECRET]";
|
||||
|
||||
export interface RedactOptions {
|
||||
maxDepth: number;
|
||||
maxArrayItems: number;
|
||||
maxObjectKeys: number;
|
||||
maxStringLength: number;
|
||||
}
|
||||
|
||||
function defaultOptions(): RedactOptions {
|
||||
const limits = getLimits();
|
||||
return {
|
||||
maxDepth: limits.maxDepth,
|
||||
maxArrayItems: limits.maxArrayItems,
|
||||
maxObjectKeys: limits.maxObjectKeys,
|
||||
maxStringLength: limits.maxString,
|
||||
};
|
||||
}
|
||||
|
||||
const SECRET_ASSIGNMENT_RE =
|
||||
/\b([A-Z0-9_]*(?:SECRET|TOKEN|PASSWORD|PASS|API[_-]?KEY|PRIVATE[_-]?KEY|AUTH|COOKIE)[A-Z0-9_]*)\s*=\s*([^\s"'`]+)/gi;
|
||||
const PRIVATE_KEY_RE = /-----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]*?-----END [A-Z ]*PRIVATE KEY-----/g;
|
||||
const BEARER_RE = /\bBearer\s+[A-Za-z0-9._~+/=-]{12,}/gi;
|
||||
const KNOWN_TOKEN_RE =
|
||||
/\b(?:sk-(?:lf|ant|proj|live|test)[A-Za-z0-9_-]*|pk-lf-[A-Za-z0-9_-]+|gh[pousr]_[A-Za-z0-9_]{20,}|npm_[A-Za-z0-9_-]{20,}|AKIA[0-9A-Z]{16})\b/g;
|
||||
const ABSOLUTE_PATH_RE =
|
||||
/(?:\/Users\/[^/\s]+|\/home\/[^/\s]+|\/private\/tmp|\/tmp|[A-Za-z]:\\Users\\[^\\\s]+)(?:[^\s"'`]*)/g;
|
||||
const SENSITIVE_FIELD_RE =
|
||||
/^(authorization|cookie|setcookie|xapikey|apikey|token|accesstoken|refreshtoken|secret|secretkey|password|passwd|privatekey)$/;
|
||||
|
||||
export function hashPath(path: string): string {
|
||||
return `[PATH_HASH:${createHash("sha256").update(path).digest("hex").slice(0, 12)}]`;
|
||||
}
|
||||
|
||||
function truncate(value: string, maxStringLength: number): string {
|
||||
return value.length > maxStringLength ? `${value.slice(0, maxStringLength)}... [truncated]` : value;
|
||||
}
|
||||
|
||||
export function redactString(value: string, options: Partial<RedactOptions> = {}): string {
|
||||
const merged = { ...defaultOptions(), ...options };
|
||||
const truncated = truncate(value, merged.maxStringLength);
|
||||
return truncated
|
||||
.replace(PRIVATE_KEY_RE, REDACTED)
|
||||
.replace(BEARER_RE, REDACTED)
|
||||
.replace(KNOWN_TOKEN_RE, REDACTED)
|
||||
.replace(SECRET_ASSIGNMENT_RE, (_match, key: string) => `${key}=${REDACTED}`)
|
||||
.replace(ABSOLUTE_PATH_RE, (path: string) => {
|
||||
const envSuffix = path.match(/([/\\]\.env(?:\.[A-Za-z0-9_-]+)?)$/)?.[1];
|
||||
return `${hashPath(envSuffix ? path.slice(0, -envSuffix.length) : path)}${envSuffix ?? ""}`;
|
||||
});
|
||||
}
|
||||
|
||||
function visit(value: unknown, options: RedactOptions, depth: number, seen: WeakSet<object>): unknown {
|
||||
if (value === null || value === undefined || typeof value === "number" || typeof value === "boolean") {
|
||||
return value;
|
||||
}
|
||||
|
||||
if (typeof value === "bigint") {
|
||||
return value.toString();
|
||||
}
|
||||
|
||||
if (typeof value === "string") {
|
||||
return redactString(value, options);
|
||||
}
|
||||
|
||||
if (typeof value === "function" || typeof value === "symbol") {
|
||||
return `[${typeof value}]`;
|
||||
}
|
||||
|
||||
if (depth <= 0) {
|
||||
return `[max depth ${options.maxDepth} reached]`;
|
||||
}
|
||||
|
||||
if (value instanceof Error) {
|
||||
return {
|
||||
name: redactString(value.name, options),
|
||||
message: redactString(value.message, options),
|
||||
stack: value.stack ? redactString(value.stack, options) : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
if (typeof value !== "object") {
|
||||
return redactString(String(value), options);
|
||||
}
|
||||
|
||||
if (seen.has(value)) {
|
||||
return "[circular]";
|
||||
}
|
||||
seen.add(value);
|
||||
|
||||
if (Array.isArray(value)) {
|
||||
const output = value
|
||||
.slice(0, options.maxArrayItems)
|
||||
.map((item) => visit(item, options, depth - 1, seen));
|
||||
if (value.length > options.maxArrayItems) {
|
||||
output.push(`[${value.length - options.maxArrayItems} truncated items]`);
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
const entries = Object.entries(value as Record<string, unknown>);
|
||||
const output: Record<string, unknown> = {};
|
||||
for (const [key, item] of entries.slice(0, options.maxObjectKeys)) {
|
||||
const normalizedKey = key.replace(/[^a-zA-Z0-9]/g, "").toLowerCase();
|
||||
output[key] = SENSITIVE_FIELD_RE.test(normalizedKey) ? REDACTED : visit(item, options, depth - 1, seen);
|
||||
}
|
||||
if (entries.length > options.maxObjectKeys) {
|
||||
output.__truncatedKeys = entries.length - options.maxObjectKeys;
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
export function redactValue(value: unknown, options: Partial<RedactOptions> = {}): unknown {
|
||||
const merged: RedactOptions = { ...defaultOptions(), ...options };
|
||||
return visit(value, merged, merged.maxDepth, new WeakSet<object>());
|
||||
}
|
||||
160
extensions/pi-langfuse/src/source-metadata.ts
Normal file
160
extensions/pi-langfuse/src/source-metadata.ts
Normal file
@@ -0,0 +1,160 @@
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { basename, dirname, join } from "node:path";
|
||||
import { execFileSync } from "node:child_process";
|
||||
|
||||
export type SourceMetadata = Record<string, string>;
|
||||
|
||||
const OVERRIDE_KEYS = new Set([
|
||||
"repo_identity",
|
||||
"repo_owner",
|
||||
"repo_name",
|
||||
"source_type",
|
||||
"service_name",
|
||||
"project_slug",
|
||||
"environment",
|
||||
"observability_owner",
|
||||
]);
|
||||
|
||||
function nonGitMetadata(): SourceMetadata {
|
||||
return {
|
||||
source_type: "non-git",
|
||||
metadata_source: "non-git",
|
||||
};
|
||||
}
|
||||
|
||||
function runGit(cwd: string, args: string[]): string {
|
||||
return execFileSync("git", args, {
|
||||
cwd,
|
||||
encoding: "utf8",
|
||||
stdio: ["ignore", "pipe", "ignore"],
|
||||
timeout: 2000,
|
||||
}).trim();
|
||||
}
|
||||
|
||||
function firstLine(value: string): string {
|
||||
return value.split(/\r?\n/).map((line) => line.trim()).find(Boolean) ?? "";
|
||||
}
|
||||
|
||||
export function sanitizeGitRemote(remoteUrl: string): Partial<Pick<SourceMetadata, "git_remote_host" | "git_remote_path">> {
|
||||
const raw = firstLine(remoteUrl).replace(/\.git$/i, "");
|
||||
if (!raw) {
|
||||
return {};
|
||||
}
|
||||
|
||||
try {
|
||||
const parsed = new URL(raw);
|
||||
const path = parsed.pathname.replace(/^\/+/, "").replace(/\.git$/i, "");
|
||||
return parsed.hostname && path ? { git_remote_host: parsed.hostname, git_remote_path: path } : {};
|
||||
} catch {
|
||||
// Continue with scp-like SSH syntax, for example git@github.com:owner/repo.git.
|
||||
}
|
||||
|
||||
const scpLike = raw.match(/^(?:[^@\s/:]+@)?([^:\s]+):(.+)$/);
|
||||
if (scpLike) {
|
||||
const host = scpLike[1];
|
||||
const path = scpLike[2].replace(/^\/+/, "").replace(/\.git$/i, "");
|
||||
return host && path && !path.includes("@") ? { git_remote_host: host, git_remote_path: path } : {};
|
||||
}
|
||||
|
||||
return {};
|
||||
}
|
||||
|
||||
function deriveIdentity(remotePath: string | undefined): Partial<Pick<SourceMetadata, "repo_identity" | "repo_owner" | "repo_name">> {
|
||||
if (!remotePath) {
|
||||
return {};
|
||||
}
|
||||
const parts = remotePath.split("/").filter(Boolean);
|
||||
if (parts.length < 2) {
|
||||
return {};
|
||||
}
|
||||
const repoName = parts[parts.length - 1];
|
||||
const owner = parts[parts.length - 2];
|
||||
if (!repoName || !owner || repoName.includes("/")) {
|
||||
return {};
|
||||
}
|
||||
return {
|
||||
repo_identity: `${owner}/${repoName}`,
|
||||
repo_owner: owner,
|
||||
repo_name: repoName,
|
||||
};
|
||||
}
|
||||
|
||||
function findRepoMetadataFile(cwd: string, gitRoot: string): string | undefined {
|
||||
let current = cwd;
|
||||
const root = gitRoot;
|
||||
while (true) {
|
||||
const candidate = join(current, ".pi-langfuse.metadata.json");
|
||||
if (existsSync(candidate)) {
|
||||
return candidate;
|
||||
}
|
||||
if (current === root) {
|
||||
break;
|
||||
}
|
||||
const parent = dirname(current);
|
||||
if (parent === current) {
|
||||
break;
|
||||
}
|
||||
current = parent;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function readWhitelistedOverrides(cwd: string, gitRoot: string): SourceMetadata {
|
||||
const path = findRepoMetadataFile(cwd, gitRoot);
|
||||
if (!path) {
|
||||
return {};
|
||||
}
|
||||
try {
|
||||
const parsed = JSON.parse(readFileSync(path, "utf8")) as Record<string, unknown>;
|
||||
const output: SourceMetadata = {};
|
||||
for (const [key, value] of Object.entries(parsed)) {
|
||||
if (OVERRIDE_KEYS.has(key) && typeof value === "string" && value.trim()) {
|
||||
output[key] = value.trim();
|
||||
}
|
||||
}
|
||||
if (output.repo_name?.includes("/")) {
|
||||
delete output.repo_name;
|
||||
}
|
||||
return output;
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
export function collectSourceMetadata(cwd: string): SourceMetadata {
|
||||
try {
|
||||
const inside = runGit(cwd, ["rev-parse", "--is-inside-work-tree"]);
|
||||
if (inside !== "true") {
|
||||
return nonGitMetadata();
|
||||
}
|
||||
|
||||
const gitRoot = runGit(cwd, ["rev-parse", "--show-toplevel"]);
|
||||
const commit = runGit(cwd, ["rev-parse", "HEAD"]);
|
||||
const branch = runGit(cwd, ["branch", "--show-current"]);
|
||||
const remote = runGit(cwd, ["config", "--get", "remote.origin.url"]);
|
||||
const remoteMetadata = sanitizeGitRemote(remote);
|
||||
const derivedIdentity = deriveIdentity(remoteMetadata.git_remote_path);
|
||||
const overrides = readWhitelistedOverrides(cwd, gitRoot);
|
||||
|
||||
const metadata: SourceMetadata = {
|
||||
source_type: "git-repo",
|
||||
repo_root_name: basename(gitRoot),
|
||||
...(branch ? { git_branch: branch } : {}),
|
||||
git_commit: commit,
|
||||
...remoteMetadata,
|
||||
...derivedIdentity,
|
||||
...overrides,
|
||||
metadata_source: Object.keys(overrides).length > 0 ? "repo-file" : "git-detection",
|
||||
};
|
||||
|
||||
if (metadata.repo_name?.includes("/")) {
|
||||
delete metadata.repo_name;
|
||||
}
|
||||
if (!metadata.repo_identity && metadata.repo_owner && metadata.repo_name) {
|
||||
metadata.repo_identity = `${metadata.repo_owner}/${metadata.repo_name}`;
|
||||
}
|
||||
return metadata;
|
||||
} catch {
|
||||
return nonGitMetadata();
|
||||
}
|
||||
}
|
||||
164
extensions/pi-langfuse/src/state.ts
Normal file
164
extensions/pi-langfuse/src/state.ts
Normal file
@@ -0,0 +1,164 @@
|
||||
import { AsyncLocalStorage } from "node:async_hooks";
|
||||
import type { Config, AgentState } from "./types.js";
|
||||
|
||||
export interface SessionRunState {
|
||||
currentModel: string;
|
||||
currentProvider: string;
|
||||
agentState: AgentState | null;
|
||||
toolCallCount: number;
|
||||
errorCount: number;
|
||||
turnCount: number;
|
||||
tracingDisabled: boolean;
|
||||
setupAttemptedThisSession: boolean;
|
||||
}
|
||||
|
||||
const DEFAULT_SESSION_ID = "__pi_langfuse_default_session__";
|
||||
|
||||
let activeSessionId = DEFAULT_SESSION_ID;
|
||||
const sessionScope = new AsyncLocalStorage<string>();
|
||||
|
||||
function createSessionRunState(): SessionRunState {
|
||||
return {
|
||||
currentModel: "",
|
||||
currentProvider: "",
|
||||
agentState: null,
|
||||
toolCallCount: 0,
|
||||
errorCount: 0,
|
||||
turnCount: 0,
|
||||
tracingDisabled: false,
|
||||
setupAttemptedThisSession: false,
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeSessionId(sessionId?: string) {
|
||||
return sessionId || DEFAULT_SESSION_ID;
|
||||
}
|
||||
|
||||
function getActiveSessionId() {
|
||||
return sessionScope.getStore() ?? activeSessionId;
|
||||
}
|
||||
|
||||
export function getSessionRunState(sessionId = getActiveSessionId()): SessionRunState {
|
||||
const normalizedSessionId = normalizeSessionId(sessionId);
|
||||
let sessionState = state.sessionStates.get(normalizedSessionId);
|
||||
if (!sessionState) {
|
||||
sessionState = createSessionRunState();
|
||||
state.sessionStates.set(normalizedSessionId, sessionState);
|
||||
}
|
||||
return sessionState;
|
||||
}
|
||||
|
||||
export function setCurrentSession(sessionId?: string) {
|
||||
activeSessionId = normalizeSessionId(sessionId);
|
||||
getSessionRunState(activeSessionId);
|
||||
}
|
||||
|
||||
export function runWithSession<T>(sessionId: string | undefined, fn: () => T): T {
|
||||
const normalizedSessionId = normalizeSessionId(sessionId);
|
||||
setCurrentSession(normalizedSessionId);
|
||||
return sessionScope.run(normalizedSessionId, fn);
|
||||
}
|
||||
|
||||
export const state = {
|
||||
config: null as Config | null,
|
||||
sessionStates: new Map<string, SessionRunState>(),
|
||||
|
||||
get currentSessionId() {
|
||||
const sessionId = getActiveSessionId();
|
||||
return sessionId === DEFAULT_SESSION_ID ? "" : sessionId;
|
||||
},
|
||||
set currentSessionId(sessionId: string) {
|
||||
setCurrentSession(sessionId);
|
||||
},
|
||||
|
||||
get currentModel() {
|
||||
return getSessionRunState().currentModel;
|
||||
},
|
||||
set currentModel(model: string) {
|
||||
getSessionRunState().currentModel = model;
|
||||
},
|
||||
|
||||
get currentProvider() {
|
||||
return getSessionRunState().currentProvider;
|
||||
},
|
||||
set currentProvider(provider: string) {
|
||||
getSessionRunState().currentProvider = provider;
|
||||
},
|
||||
|
||||
get agentState() {
|
||||
return getSessionRunState().agentState;
|
||||
},
|
||||
set agentState(agentState: AgentState | null) {
|
||||
getSessionRunState().agentState = agentState;
|
||||
},
|
||||
|
||||
get toolCallCount() {
|
||||
return getSessionRunState().toolCallCount;
|
||||
},
|
||||
set toolCallCount(toolCallCount: number) {
|
||||
getSessionRunState().toolCallCount = toolCallCount;
|
||||
},
|
||||
|
||||
get errorCount() {
|
||||
return getSessionRunState().errorCount;
|
||||
},
|
||||
set errorCount(errorCount: number) {
|
||||
getSessionRunState().errorCount = errorCount;
|
||||
},
|
||||
|
||||
get turnCount() {
|
||||
return getSessionRunState().turnCount;
|
||||
},
|
||||
set turnCount(turnCount: number) {
|
||||
getSessionRunState().turnCount = turnCount;
|
||||
},
|
||||
|
||||
get isTracingDisabled() {
|
||||
return getSessionRunState().tracingDisabled;
|
||||
},
|
||||
set isTracingDisabled(disabled: boolean) {
|
||||
getSessionRunState().tracingDisabled = disabled;
|
||||
},
|
||||
|
||||
get setupAttemptedThisSession() {
|
||||
return getSessionRunState().setupAttemptedThisSession;
|
||||
},
|
||||
set setupAttemptedThisSession(attempted: boolean) {
|
||||
getSessionRunState().setupAttemptedThisSession = attempted;
|
||||
},
|
||||
};
|
||||
|
||||
export function resetRunState(sessionId = getActiveSessionId()) {
|
||||
const normalizedSessionId = normalizeSessionId(sessionId);
|
||||
const setupAttemptedThisSession =
|
||||
state.sessionStates.get(normalizedSessionId)?.setupAttemptedThisSession ?? false;
|
||||
state.sessionStates.set(normalizedSessionId, {
|
||||
...createSessionRunState(),
|
||||
setupAttemptedThisSession,
|
||||
});
|
||||
}
|
||||
|
||||
export function clearAllSessionStates() {
|
||||
state.sessionStates.clear();
|
||||
activeSessionId = DEFAULT_SESSION_ID;
|
||||
getSessionRunState();
|
||||
}
|
||||
|
||||
export function computeEvaluationScores(sessionId = getActiveSessionId()) {
|
||||
const sessionState = getSessionRunState(sessionId);
|
||||
const toolSuccessRate =
|
||||
sessionState.toolCallCount > 0
|
||||
? (sessionState.toolCallCount - sessionState.errorCount) / sessionState.toolCallCount
|
||||
: 1;
|
||||
const sessionHadErrors = sessionState.errorCount > 0;
|
||||
|
||||
return {
|
||||
tool_call_count: sessionState.toolCallCount,
|
||||
turn_count: sessionState.turnCount,
|
||||
total_tool_errors: sessionState.errorCount,
|
||||
tool_success_rate: toolSuccessRate,
|
||||
session_had_errors: sessionHadErrors ? 1 : 0,
|
||||
};
|
||||
}
|
||||
|
||||
getSessionRunState();
|
||||
139
extensions/pi-langfuse/src/types.ts
Normal file
139
extensions/pi-langfuse/src/types.ts
Normal file
@@ -0,0 +1,139 @@
|
||||
import type { CapturePolicy } from "./capture-policy.js";
|
||||
import type { PayloadLimits } from "./limits.js";
|
||||
|
||||
export interface Config {
|
||||
publicKey: string;
|
||||
secretKey: string;
|
||||
host: string;
|
||||
capturePolicy?: CapturePolicy;
|
||||
limits?: PayloadLimits;
|
||||
}
|
||||
|
||||
export interface LangfuseObservation {
|
||||
id?: string;
|
||||
traceId?: string;
|
||||
update(body?: ObservationUpdate): LangfuseObservation;
|
||||
end(body?: ObservationUpdate): void;
|
||||
startObservation?(
|
||||
name: string,
|
||||
body?: ObservationUpdate,
|
||||
options?: { asType?: "agent" | "generation" | "tool" | "span" },
|
||||
): LangfuseObservation;
|
||||
setTraceIO?(body?: { input?: unknown; output?: unknown }): void;
|
||||
}
|
||||
|
||||
export interface ObservationUpdate {
|
||||
input?: unknown;
|
||||
output?: unknown;
|
||||
metadata?: Record<string, unknown>;
|
||||
model?: string;
|
||||
modelParameters?: Record<string, string | number>;
|
||||
usageDetails?: Record<string, number>;
|
||||
usage?: Record<string, number>;
|
||||
costDetails?: Record<string, number>;
|
||||
level?: "DEBUG" | "DEFAULT" | "WARNING" | "ERROR";
|
||||
statusMessage?: string;
|
||||
completionStartTime?: Date;
|
||||
}
|
||||
|
||||
export interface LangfuseScoreClient {
|
||||
api?: {
|
||||
trace?: {
|
||||
get?: (traceId: string) => Promise<unknown>;
|
||||
};
|
||||
ingestion?: {
|
||||
batch?: (request: unknown) => Promise<unknown>;
|
||||
};
|
||||
};
|
||||
score?: {
|
||||
create(body: {
|
||||
traceId?: string;
|
||||
sessionId?: string;
|
||||
observationId?: string;
|
||||
name: string;
|
||||
value: number;
|
||||
dataType?: "NUMERIC" | "BOOLEAN";
|
||||
}): unknown;
|
||||
};
|
||||
flush?: () => Promise<void>;
|
||||
shutdown?: () => Promise<void>;
|
||||
}
|
||||
|
||||
export interface PendingScore {
|
||||
traceId?: string;
|
||||
sessionId?: string;
|
||||
observationId?: string;
|
||||
name: string;
|
||||
value: number;
|
||||
dataType?: "NUMERIC" | "BOOLEAN";
|
||||
environment?: string;
|
||||
}
|
||||
|
||||
export interface LangfuseRuntimeConfig {
|
||||
publicKey: string;
|
||||
secretKey: string;
|
||||
host: string;
|
||||
}
|
||||
|
||||
export interface LangfuseRuntime {
|
||||
startObservation: (
|
||||
name: string,
|
||||
body?: ObservationUpdate,
|
||||
options?: { asType?: "agent" | "generation" | "tool" | "span" },
|
||||
) => LangfuseObservation;
|
||||
propagateAttributes: (
|
||||
params: {
|
||||
sessionId?: string;
|
||||
traceName?: string;
|
||||
metadata?: Record<string, string>;
|
||||
tags?: string[];
|
||||
},
|
||||
fn: () => LangfuseObservation,
|
||||
) => LangfuseObservation;
|
||||
scoreClient: LangfuseScoreClient;
|
||||
spanProcessor?: { forceFlush?: () => Promise<void>; shutdown?: () => Promise<void> };
|
||||
tracerProvider?: { forceFlush?: () => Promise<void>; shutdown?: () => Promise<void> };
|
||||
clearTracerProvider?: () => void;
|
||||
restFallback?: unknown;
|
||||
pendingScores?: PendingScore[];
|
||||
scoreFlushAt?: number;
|
||||
scoreFlushIntervalMs?: number;
|
||||
scoreRequestTimeoutMs?: number;
|
||||
scoreFlushTimer?: NodeJS.Timeout;
|
||||
scoreFlushPromise?: Promise<void>;
|
||||
scoreFlushController?: AbortController;
|
||||
scoreFlushStopped?: boolean;
|
||||
runtimeConfig?: LangfuseRuntimeConfig;
|
||||
}
|
||||
|
||||
export interface GenerationState {
|
||||
observation: LangfuseObservation;
|
||||
requestKey: string;
|
||||
ended: boolean;
|
||||
metadata: Record<string, unknown>;
|
||||
modelParameters?: Record<string, string | number>;
|
||||
ttftRecorded?: boolean;
|
||||
}
|
||||
|
||||
export interface ToolState {
|
||||
observation: LangfuseObservation;
|
||||
toolName: string;
|
||||
ended: boolean;
|
||||
startedAt: number;
|
||||
inputBytes: number;
|
||||
}
|
||||
|
||||
export interface AgentState {
|
||||
root?: LangfuseObservation;
|
||||
activeTurn?: LangfuseObservation;
|
||||
traceId?: string;
|
||||
promptInput?: unknown;
|
||||
cwd?: string;
|
||||
generationSeq: number;
|
||||
activeGenerations: Map<string, GenerationState>;
|
||||
generationOrder: string[];
|
||||
activeTools: Map<string, ToolState>;
|
||||
latestAssistantOutput?: unknown;
|
||||
sourceMetadata?: Record<string, unknown>;
|
||||
providerMetadataByRequest: Map<string, Record<string, unknown>>;
|
||||
}
|
||||
427
extensions/pi-langfuse/src/utils.ts
Normal file
427
extensions/pi-langfuse/src/utils.ts
Normal file
@@ -0,0 +1,427 @@
|
||||
import { getLimits } from "./limits.js";
|
||||
import { createCapturePolicy, type CapturePolicy } from "./capture-policy.js";
|
||||
import { redactValue } from "./redaction.js";
|
||||
import { state } from "./state.js";
|
||||
|
||||
export function getCapturePolicy(): CapturePolicy {
|
||||
return state.config?.capturePolicy ?? createCapturePolicy();
|
||||
}
|
||||
|
||||
export { getLimits };
|
||||
|
||||
export function truncate(value: string, maxLength = getLimits().maxString): string {
|
||||
return value.length > maxLength ? `${value.slice(0, maxLength)}... [truncated]` : value;
|
||||
}
|
||||
|
||||
export function tryParseJson(value: string): unknown {
|
||||
const trimmed = value.trim();
|
||||
if (!trimmed || !["{", "["].includes(trimmed[0])) {
|
||||
return value;
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(trimmed);
|
||||
} catch {
|
||||
return value;
|
||||
}
|
||||
}
|
||||
|
||||
const PAYLOAD_TOO_LARGE = "[payload too large]";
|
||||
|
||||
export function shapePayload(
|
||||
value: unknown,
|
||||
options: {
|
||||
maxString?: number;
|
||||
depth?: number;
|
||||
maxNodes?: number;
|
||||
maxArrayItems?: number;
|
||||
maxObjectKeys?: number;
|
||||
redact?: boolean;
|
||||
parseJson?: boolean;
|
||||
} = {},
|
||||
): unknown {
|
||||
const limits = getLimits();
|
||||
const maxString = options.maxString ?? limits.maxString;
|
||||
const depth = options.depth ?? limits.maxDepth;
|
||||
const maxNodes = options.maxNodes ?? limits.maxNodes;
|
||||
const maxArrayItems = options.maxArrayItems ?? limits.maxArrayItems;
|
||||
const maxObjectKeys = options.maxObjectKeys ?? limits.maxObjectKeys;
|
||||
const budget = { exhausted: false, nodeCount: 0 };
|
||||
|
||||
function visit(item: unknown, remainingDepth: number, seen: WeakSet<object>): unknown {
|
||||
if (budget.exhausted) {
|
||||
return PAYLOAD_TOO_LARGE;
|
||||
}
|
||||
|
||||
budget.nodeCount++;
|
||||
if (budget.nodeCount > maxNodes) {
|
||||
budget.exhausted = true;
|
||||
return PAYLOAD_TOO_LARGE;
|
||||
}
|
||||
|
||||
if (typeof item === "string") {
|
||||
const truncated = truncate(item, maxString);
|
||||
if (options.parseJson === false) {
|
||||
return truncated;
|
||||
}
|
||||
const parsed = tryParseJson(truncated);
|
||||
if (parsed === truncated) {
|
||||
return truncated;
|
||||
}
|
||||
return visit(parsed, remainingDepth - 1, seen);
|
||||
}
|
||||
|
||||
if (
|
||||
item === null ||
|
||||
typeof item === "undefined" ||
|
||||
typeof item === "number" ||
|
||||
typeof item === "boolean"
|
||||
) {
|
||||
return item;
|
||||
}
|
||||
|
||||
if (typeof item === "bigint") {
|
||||
return item.toString();
|
||||
}
|
||||
|
||||
if (typeof item === "function" || typeof item === "symbol") {
|
||||
return `[${typeof item}]`;
|
||||
}
|
||||
|
||||
if (remainingDepth <= 0) {
|
||||
return `[max depth ${depth} reached]`;
|
||||
}
|
||||
|
||||
if (Array.isArray(item)) {
|
||||
const output: unknown[] = [];
|
||||
const limit = Math.min(item.length, maxArrayItems);
|
||||
for (let index = 0; index < limit; index++) {
|
||||
output.push(visit(item[index], remainingDepth - 1, seen));
|
||||
if (budget.exhausted) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
if (item instanceof Error) {
|
||||
return {
|
||||
name: item.name,
|
||||
message: item.message,
|
||||
stack: item.stack ? truncate(item.stack, maxString) : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
if (typeof item === "object") {
|
||||
if (seen.has(item)) {
|
||||
return "[circular]";
|
||||
}
|
||||
seen.add(item);
|
||||
|
||||
const output: Record<string, unknown> = {};
|
||||
let keyCount = 0;
|
||||
for (const key in item as Record<string, unknown>) {
|
||||
if (!Object.hasOwn(item, key)) {
|
||||
continue;
|
||||
}
|
||||
output[key] = visit((item as Record<string, unknown>)[key], remainingDepth - 1, seen);
|
||||
keyCount++;
|
||||
if (budget.exhausted || keyCount >= maxObjectKeys) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
return String(item);
|
||||
}
|
||||
|
||||
const shaped = visit(value, depth, new WeakSet<object>());
|
||||
return options.redact === false
|
||||
? shaped
|
||||
: redactValue(shaped, {
|
||||
maxDepth: depth,
|
||||
maxStringLength: maxString,
|
||||
maxArrayItems,
|
||||
maxObjectKeys,
|
||||
});
|
||||
}
|
||||
|
||||
export function safeSerialize(value: unknown, maxLength = getLimits().maxToolPayload): string {
|
||||
try {
|
||||
return truncate(JSON.stringify(shapePayload(value, { maxString: maxLength }), null, 2), maxLength);
|
||||
} catch {
|
||||
return `[unserializable ${typeof value}]`;
|
||||
}
|
||||
}
|
||||
|
||||
export function estimatePayloadBytes(value: unknown, maxLength = getLimits().maxToolPayload): number {
|
||||
return new TextEncoder().encode(safeSerialize(value, maxLength)).length;
|
||||
}
|
||||
|
||||
export function extractTextContent(content: unknown, maxLength?: number): string | undefined {
|
||||
if (typeof content === "string") {
|
||||
return maxLength ? truncate(content, maxLength) : content;
|
||||
}
|
||||
|
||||
if (!Array.isArray(content)) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const text = content
|
||||
.map((item) => {
|
||||
if (!item || typeof item !== "object") return "";
|
||||
const block = item as { type?: string; text?: string; thinking?: string };
|
||||
return block.type === "text" && block.text ? block.text : "";
|
||||
})
|
||||
.filter(Boolean)
|
||||
.join("\n");
|
||||
|
||||
if (!text) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return maxLength ? truncate(text, maxLength) : text;
|
||||
}
|
||||
|
||||
export function normalizeContentForLangfuse(content: unknown, api?: string): unknown {
|
||||
if (!Array.isArray(content)) {
|
||||
return content;
|
||||
}
|
||||
|
||||
const toolCallItems = content.filter((item) => {
|
||||
return item && typeof item === "object" && (item as { type?: string }).type === "toolCall";
|
||||
});
|
||||
if (toolCallItems.length === 0) {
|
||||
return content;
|
||||
}
|
||||
|
||||
const text = content
|
||||
.map((item) => {
|
||||
if (!item || typeof item !== "object") return "";
|
||||
const block = item as { type?: string; text?: string };
|
||||
return block.type === "text" && typeof block.text === "string" ? block.text : "";
|
||||
})
|
||||
.filter(Boolean)
|
||||
.join("");
|
||||
|
||||
if (api === "anthropic-messages") {
|
||||
const blocks: unknown[] = [];
|
||||
if (text) {
|
||||
blocks.push({ type: "text", text });
|
||||
}
|
||||
for (const item of toolCallItems) {
|
||||
const toolCall = item as { id?: unknown; name?: unknown; arguments?: unknown };
|
||||
const toolInput = shapePayload(toolCall.arguments, { parseJson: false });
|
||||
blocks.push({
|
||||
type: "tool_use",
|
||||
id: String(toolCall.id ?? ""),
|
||||
name: String(toolCall.name ?? "tool"),
|
||||
input: toolInput,
|
||||
});
|
||||
}
|
||||
return blocks;
|
||||
}
|
||||
|
||||
return {
|
||||
role: "assistant",
|
||||
content: text || null,
|
||||
tool_calls: toolCallItems.map((item) => {
|
||||
const toolCall = item as { id?: unknown; name?: unknown; arguments?: unknown };
|
||||
const toolArguments = shapePayload(toolCall.arguments ?? {}, { parseJson: false });
|
||||
return {
|
||||
id: String(toolCall.id ?? ""),
|
||||
type: "function",
|
||||
function: {
|
||||
name: String(toolCall.name ?? "tool"),
|
||||
arguments: typeof toolArguments === "string" ? toolArguments : JSON.stringify(toolArguments),
|
||||
},
|
||||
};
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
export function extractToolCalls(message: Record<string, unknown>): unknown | undefined {
|
||||
return (
|
||||
message.toolCalls ??
|
||||
message.tool_calls ??
|
||||
message.function_calls ??
|
||||
(message.content && Array.isArray(message.content)
|
||||
? message.content.filter((block) => {
|
||||
return block && typeof block === "object" && ["tool_use", "tool_call", "toolCall"].includes(String((block as { type?: string }).type));
|
||||
})
|
||||
: undefined)
|
||||
);
|
||||
}
|
||||
|
||||
export function extractAssistantOutput(message: unknown): unknown | undefined {
|
||||
if (!message || typeof message !== "object") {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const msg = message as Record<string, unknown>;
|
||||
const normalizedContent = normalizeContentForLangfuse(msg.content, typeof msg.api === "string" ? msg.api : undefined);
|
||||
if (normalizedContent !== msg.content) {
|
||||
return shapePayload(normalizedContent, { parseJson: false });
|
||||
}
|
||||
|
||||
const text = extractTextContent(msg.content);
|
||||
if (text) {
|
||||
return text;
|
||||
}
|
||||
|
||||
const toolCalls = extractToolCalls(msg);
|
||||
if (Array.isArray(toolCalls) && toolCalls.length > 0) {
|
||||
return { toolCalls: shapePayload(toolCalls) };
|
||||
}
|
||||
|
||||
if (toolCalls) {
|
||||
return { toolCalls: shapePayload(toolCalls) };
|
||||
}
|
||||
|
||||
return shapePayload(msg);
|
||||
}
|
||||
|
||||
export function extractFinalAssistant(messages: unknown): Record<string, unknown> | undefined {
|
||||
if (!Array.isArray(messages)) {
|
||||
return undefined;
|
||||
}
|
||||
return messages.filter((message) => message?.role === "assistant").pop() as Record<string, unknown> | undefined;
|
||||
}
|
||||
|
||||
export function getRequestKey(event: Record<string, unknown>, fallback: string): string {
|
||||
return String(
|
||||
event.requestId ??
|
||||
event.providerRequestId ??
|
||||
event.messageId ??
|
||||
event.turnId ??
|
||||
event.turnIndex ??
|
||||
event.id ??
|
||||
fallback,
|
||||
);
|
||||
}
|
||||
|
||||
export function getToolCallId(event: Record<string, unknown>): string | undefined {
|
||||
const id = event.toolCallId ?? event.id ?? event.callId ?? event.tool_use_id ?? event.toolUseId;
|
||||
return id === undefined || id === null ? undefined : String(id);
|
||||
}
|
||||
|
||||
export function getToolName(event: Record<string, unknown>): string {
|
||||
return String(
|
||||
event.toolName ??
|
||||
event.name ??
|
||||
event.tool ??
|
||||
event.functionName ??
|
||||
(event.call && typeof event.call === "object" ? (event.call as Record<string, unknown>).name : undefined) ??
|
||||
"tool",
|
||||
);
|
||||
}
|
||||
|
||||
export function getToolInput(event: Record<string, unknown>): unknown {
|
||||
return (
|
||||
event.input ??
|
||||
event.args ??
|
||||
event.arguments ??
|
||||
event.params ??
|
||||
(event.call && typeof event.call === "object" ? (event.call as Record<string, unknown>).input : undefined) ??
|
||||
event
|
||||
);
|
||||
}
|
||||
|
||||
export function getProviderPayload(event: Record<string, unknown>): unknown {
|
||||
return event.request ?? event.payload ?? event.body ?? event.providerPayload ?? event.messages ?? event;
|
||||
}
|
||||
|
||||
export function extractModelParameters(payload: unknown): Record<string, string | number> | undefined {
|
||||
if (!payload || typeof payload !== "object" || Array.isArray(payload)) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const params: Record<string, string | number> = {};
|
||||
const record = payload as Record<string, unknown>;
|
||||
for (const key of [
|
||||
"temperature",
|
||||
"top_p",
|
||||
"topP",
|
||||
"max_tokens",
|
||||
"maxTokens",
|
||||
"max_completion_tokens",
|
||||
"presence_penalty",
|
||||
"frequency_penalty",
|
||||
"reasoning_effort",
|
||||
]) {
|
||||
const value = record[key];
|
||||
if (typeof value === "string" || typeof value === "number") {
|
||||
params[key] = value;
|
||||
}
|
||||
}
|
||||
|
||||
return Object.keys(params).length > 0 ? params : undefined;
|
||||
}
|
||||
|
||||
export function getMessageFromEvent(event: Record<string, unknown>): Record<string, unknown> | undefined {
|
||||
if (event.message && typeof event.message === "object") {
|
||||
return event.message as Record<string, unknown>;
|
||||
}
|
||||
if (event.role || event.content) {
|
||||
return event;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
export function extractUsage(messageOrEvent: Record<string, unknown>): Record<string, number> | undefined {
|
||||
const usage = (messageOrEvent.usage ??
|
||||
(messageOrEvent.message && typeof messageOrEvent.message === "object"
|
||||
? (messageOrEvent.message as Record<string, unknown>).usage
|
||||
: undefined)) as Record<string, unknown> | undefined;
|
||||
if (!usage || typeof usage !== "object") {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const input = Number(usage.input ?? usage.inputTokens ?? usage.prompt_tokens ?? usage.promptTokens ?? 0);
|
||||
const output = Number(usage.output ?? usage.outputTokens ?? usage.completion_tokens ?? usage.completionTokens ?? 0);
|
||||
const total = Number(usage.total ?? usage.totalTokens ?? usage.total_tokens ?? input + output);
|
||||
const cacheRead = Number(usage.cacheRead ?? usage.cache_read ?? usage.cachedTokens ?? 0);
|
||||
const cacheWrite = Number(usage.cacheWrite ?? usage.cache_write ?? 0);
|
||||
|
||||
return {
|
||||
input,
|
||||
output,
|
||||
total,
|
||||
...(cacheRead ? { cacheRead } : {}),
|
||||
...(cacheWrite ? { cacheWrite } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
export function extractCostDetails(messageOrEvent: Record<string, unknown>): Record<string, number> | undefined {
|
||||
const usage = (messageOrEvent.usage ??
|
||||
(messageOrEvent.message && typeof messageOrEvent.message === "object"
|
||||
? (messageOrEvent.message as Record<string, unknown>).usage
|
||||
: undefined)) as Record<string, unknown> | undefined;
|
||||
const cost = (messageOrEvent.cost ?? usage?.cost ?? messageOrEvent.costDetails) as Record<string, unknown> | undefined;
|
||||
if (!cost || typeof cost !== "object") {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const input = Number(cost.input ?? cost.inputCost ?? 0);
|
||||
const output = Number(cost.output ?? cost.outputCost ?? 0);
|
||||
const total = Number(cost.total ?? cost.totalCost ?? input + output);
|
||||
if (input === 0 && output === 0 && total === 0) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return { input, output, total };
|
||||
}
|
||||
|
||||
export function extractResponseMetadata(event: Record<string, unknown>): Record<string, unknown> {
|
||||
return shapePayload(
|
||||
{
|
||||
status: event.status ?? event.statusCode ?? event.httpStatus,
|
||||
headers: event.headers,
|
||||
responseHeaders: event.responseHeaders,
|
||||
providerMetadata: event.providerMetadata ?? event.metadata,
|
||||
requestId: event.requestId ?? event.providerRequestId,
|
||||
},
|
||||
{ depth: 4, maxString: 4_000 },
|
||||
) as Record<string, unknown>;
|
||||
}
|
||||
16
extensions/pi-langfuse/tsconfig.json
Normal file
16
extensions/pi-langfuse/tsconfig.json
Normal file
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "NodeNext",
|
||||
"moduleResolution": "NodeNext",
|
||||
"strict": true,
|
||||
"noEmit": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true
|
||||
},
|
||||
"include": [
|
||||
"index.ts",
|
||||
"src/**/*.ts",
|
||||
"types/**/*.d.ts"
|
||||
]
|
||||
}
|
||||
49
extensions/pi-langfuse/types/langfuse-runtime-shims.d.ts
vendored
Normal file
49
extensions/pi-langfuse/types/langfuse-runtime-shims.d.ts
vendored
Normal file
@@ -0,0 +1,49 @@
|
||||
declare module "@opentelemetry/sdk-trace-base" {
|
||||
export class BasicTracerProvider {
|
||||
constructor(options?: { spanProcessors?: unknown[] });
|
||||
forceFlush?(): Promise<void>;
|
||||
shutdown?(): Promise<void>;
|
||||
}
|
||||
}
|
||||
|
||||
declare module "@langfuse/otel" {
|
||||
export class LangfuseSpanProcessor {
|
||||
constructor(options: {
|
||||
publicKey: string;
|
||||
secretKey: string;
|
||||
baseUrl: string;
|
||||
});
|
||||
forceFlush?(): Promise<void>;
|
||||
shutdown?(): Promise<void>;
|
||||
}
|
||||
}
|
||||
|
||||
declare module "@langfuse/tracing" {
|
||||
export function setLangfuseTracerProvider(provider: unknown): void;
|
||||
|
||||
export function startObservation(
|
||||
name: string,
|
||||
body?: Record<string, unknown>,
|
||||
options?: { asType?: string },
|
||||
): unknown;
|
||||
|
||||
export function propagateAttributes<T>(
|
||||
params: {
|
||||
sessionId?: string;
|
||||
traceName?: string;
|
||||
metadata?: Record<string, string>;
|
||||
tags?: string[];
|
||||
},
|
||||
fn: () => T,
|
||||
): T;
|
||||
}
|
||||
|
||||
declare module "@langfuse/client" {
|
||||
export class LangfuseClient {
|
||||
constructor(options: {
|
||||
publicKey: string;
|
||||
secretKey: string;
|
||||
baseUrl: string;
|
||||
});
|
||||
}
|
||||
}
|
||||
29
extensions/pi-langfuse/types/node-shims.d.ts
vendored
Normal file
29
extensions/pi-langfuse/types/node-shims.d.ts
vendored
Normal file
@@ -0,0 +1,29 @@
|
||||
declare module "node:fs" {
|
||||
export function mkdirSync(path: string, options?: { recursive?: boolean }): void;
|
||||
export function readFileSync(path: string, encoding: string): string;
|
||||
export function existsSync(path: string): boolean;
|
||||
export function writeFileSync(path: string, data: string, encoding: string): void;
|
||||
}
|
||||
|
||||
declare module "node:crypto" {
|
||||
export function randomUUID(): string;
|
||||
}
|
||||
|
||||
declare module "node:os" {
|
||||
export function homedir(): string;
|
||||
}
|
||||
|
||||
declare module "node:path" {
|
||||
export function resolve(...paths: string[]): string;
|
||||
export function dirname(path: string): string;
|
||||
export function basename(path: string, suffix?: string): string;
|
||||
}
|
||||
|
||||
declare module "node:url" {
|
||||
export function fileURLToPath(url: string | URL): string;
|
||||
}
|
||||
|
||||
declare const process: {
|
||||
cwd(): string;
|
||||
env: Record<string, string | undefined>;
|
||||
};
|
||||
9
extensions/pi-langfuse/types/pi-coding-agent.d.ts
vendored
Normal file
9
extensions/pi-langfuse/types/pi-coding-agent.d.ts
vendored
Normal file
@@ -0,0 +1,9 @@
|
||||
declare module "@earendil-works/pi-coding-agent" {
|
||||
export interface ExtensionAPI {
|
||||
on(event: string, handler: (event: any, ctx: any) => unknown): void;
|
||||
registerCommand(
|
||||
name: string,
|
||||
options: { description?: string; handler: (args: string, ctx: any) => unknown },
|
||||
): void;
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user