dsh-telemetry

English | 简体中文

A local-first Harness telemetry plugin: records request, model, tool, and plugin lifecycle metrics (latency, tokens, cost, errors, cache) — no conversation content collected by default.

> The npm package is named dsh-local-telemetry (the dsh-telemetry name on npm is taken by a third party); the GitHub repository remains dsh-telemetry. Both refer to the same project.

Positioning

dsh-telemetry is a Harness runtime observability plugin. It does not do business analysis, does not modify request content, and does not upload user conversations to any third-party platform.

It answers:
- How long did a request take?
- Where did the time go — model, tool, plugin, or queuing?
- What are the input/output token counts and retry costs?
- Which tool calls are slowest and most failure-prone?
- Which plugins throw exceptions or block?
- Is the cache hitting, and is model routing saving cost?
- Are there oversized contexts, tool-call loops, or abnormal retries?

One-line positioning:

> Make Harness behavior measurable without collecting sensitive conversation content by default.

UI Preview

Dashboard overview — requests, P95 latency, time-to-first-token, tokens/cache hits, estimated cost (computed automatically once a price catalog is configured), and error classification. Every metric is annotated with its sample count and can be recomputed from raw events:

Tool & plugin timing — loop-call ⚠ warnings, counts of calls requiring user confirmation, and plugin hook error statistics; the request timeline below uses status dots for success/failure/cancel, with ↻ marks on retried requests:

Trace details — click any request to expand its span tree and event timeline; the screenshot below shows the full chain of deepseek-reasoner hitting rate_limit and falling back to deepseek-chat successfully (each attempt timed independently):

> All screenshots show the local read-only Web UI (--ui, bound to 127.0.0.1 only) with a demo dataset; by default no prompt / response / file content / secrets are collected.

Installation

As a DSH plugin (recommended):

dsh plugin --profile web add "github:duyanta123/dsh-telemetry#v0.2.1"

Compatibility tiers: JSONL and pure CLI capabilities run standalone on Node.js >= 18; the SQLite backend requires Node.js >= 22.5; as a DSH 0.2.1-rc.2 plugin it is verified with Node.js >= 22.19. Run npm run test:compat to execute an isolated-profile add, dump-config, a real-host session-capture smoke test (drives turn/tool traffic and asserts the adapter actually collects it), and a limited-duration startup check.

Minimum host version: automatic session capture is verified against DeepSeek Harness 0.1.5-rc.1+ through 0.2.1-rc.2 (compat gate pins 0.2.1-rc.2). On older hosts the plugin still installs and runs — it degrades to explicit-emit mode (skill + CLI + standalone UI) and prints a one-time notice; nothing else changes.

Fresh machine (DSH only, no other tooling required):

# 1. in any DSH profile
dsh plugin --profile  add "github:duyanta123/dsh-telemetry#v0.2.1"
# 2. restart that profile (e.g. dsh --profile ) — capture starts automatically
# 3. query or open the dashboard
node /node_modules/dsh-local-telemetry/bin/telemetry.mjs --status
node /node_modules/dsh-local-telemetry/bin/telemetry.mjs --ui --port 47610

Or install from npm (as a library or standalone CLI):

npm install dsh-local-telemetry

After installation, restart dsh --profile web; the telemetry-runbook skill then guides the query CLI:

node bin/telemetry.mjs --status
node bin/telemetry.mjs --summary --since 24h
node bin/telemetry.mjs --trace
node bin/telemetry.mjs --ui --port 47610

Quick Start

1. Automatic host integration (DeepSeek Harness 0.1.5-rc.1)

When a DSH profile loads this plugin, plugin/index.js subscribes to the session event bus declared by @deepseek-ai/dsh-session: session/created, session/event, session/flush, session/disposed. That bus is post-commit and fire-and-forget, and the host contains observer failures — so capture can never affect the agent loop. No host service is registered and no request semantics change.

Mapping (plan §3.1: one DSH turn = one request span):

Never read (privacy §6): tool/call.arguments, assistant/message.message.content, stream.texts, and the request/header body (it carries the full system prompt). Only names, ids, counts, statuses, durations, and token totals are recorded.

Why not the sessionTelemetry seam: dsh-base already mounts @deepseek-ai/dsh-session-telemetry-otel, and a duplicate service registration throws. That row also ships mode: FEEDBACK_ONLY with an on-demand coordinator — it captures only after explicit user feedback, so it cannot back always-on local telemetry. The session-telemetry/record waterfall is the redaction hook on that same on-demand path.

Capabilities are reported honestly via getCapabilities(): lifecycle_hooks: "partial" — usage/cache/first-token are confirmed, plugin lifecycle is partial (command hooks only; in-process plugin init is not observable on the bus), and cost always comes from the versioned price catalog rather than the host.

Configure it from the profile patch:

- id: dsh-local-telemetry
name: dsh-local-telemetry
config:
telemetry:
store: sqlite
path: ~/.dsh/telemetry
sample_rate: 1

Non-DSH hosts, tests, and higher-level plugins can still drive the recorder explicitly. Note that createRecorder takes { config, sink, now } — the plugin config goes under config:

import { createRecorder } from 'dsh-local-telemetry/telemetry';

const recorder = createRecorder({ config: { path: '~/.dsh/telemetry' } });
await recorder.start();   // async: the sqlite backend loads on demand
recorder.record({ event: 'model.completed', trace_id: 'trace-001', span_id: 'span-003' });
await recorder.close();   // flush; failures never block exit

2. Aggregate reads (referenced by higher-level plugins)

import { openStore, aggregateEvents, buildTraceView } from 'dsh-local-telemetry/telemetry';

const store = await openStore({ store: 'jsonl', path: '~/.dsh/telemetry' });
const { events } = await store.readEvents({ fromMs: Date.now() - 3600e3 });
const summary = aggregateEvents(events, { catalog: null });

console.log(`P95 latency: ${summary.requests.latency.p95}ms, Input tokens: ${summary.tokens.input}`);

3. Invoke via the DSH skill (the skill guides the CLI)

node bin/telemetry.mjs --summary --since 1h --group-by model
node bin/telemetry.mjs --export TELEMETRY-REPORT.md --since 7d --format markdown
node bin/telemetry.mjs --purge --before 30d
node bin/telemetry.mjs --ui --port 47610

CLI Options

Privacy & Security

• No content by default: prompts, responses, file contents, command arguments, environment variables, and secrets are never collected.
• Sanitization: sensitive keys (Authorization, Cookie, token, password, api_key, etc.) are dropped whole; URL credentials and query tokens are redacted; absolute paths can be reduced to basenames or hashes.
• Name hashing: tool, plugin, model, and profile names can be hashed — stable but not directly reversible.
• Local storage: defaults to ~/.dsh/telemetry (JSONL files per day, optional SQLite); no network access.
• Read-only UI: --ui binds to 127.0.0.1 only and never exposes to the LAN by default.

Troubleshooting

--store sqlite fails to start?
The SQLite backend relies on the built-in node:sqlite, which requires Node >= 22.5. Check with node --version, or switch to the default JSONL backend (Node >= 18 suffices).

Web UI unreachable, or the port is taken?
--ui binds to 127.0.0.1 only (by design, never exposed to the LAN); access remote machines through an SSH tunnel. If the default port 47610 is taken, change it with --port.

Cost shows as null in summaries?
Cost = token usage × price catalog. Without a catalog (--config or prices.json in the data directory), cost is recorded as missing (null, not 0) and can be recomputed for past events once configured; see docs/schema.md for field semantics.

--status says collection is not active?
On config parse failure the recorder degrades to disabled (fail-open, a safe default) — check the config file syntax and field names. JSONL events are written per-day under ~/.dsh/telemetry; confirm today's file exists there.

Old sessions won't open after upgrading the DSH host to 0.1.5.x?
The Session format V3 migration is irreversible and is host behavior; back up session logs before upgrading the host (see the 0.1.2 entry in CHANGELOG.md).

Documentation

• docs/configuration.md — config file, field tables, sanitization rules, resource budgets, price catalog format
• docs/schema.md — event contract schema version 1.0, 12 event types and field constraints
• examples/telemetry.json / examples/prices.json — config samples
• CHANGELOG.md — release notes
• DSH-TELEMETRY-开发计划.md — design and iteration history

Versions & Roadmap

• v0.1.0 (released): local JSONL + optional SQLite (built-in node:sqlite, Node ≥22.5), request/model/tool/plugin base events, start/stop config, fail-open, --status/--summary/--trace/--export/--purge/--ui, no content by default, sampling and retention policies, price catalog and sanitization, Markdown reports, trace/span tree and timeline views.
• v0.1.1 / v0.1.2 (released): npm run test:compat compatibility gate and three-tier compatibility notes; DSH host baseline migrated to 0.1.5-rc.2 (zero plugin code changes).
• v0.2.0: automatic host integration — subscribes to the DSH session event bus (per-session capture with session.id, requests/attempts/tools/hooks mapped from real session flow), retry-count semantics fixed to "re-issued after failure", sqlite→jsonl store fallback with effective-store reporting, and honest not_started handling instead of silent drops.
• v0.2.1: honest telemetry self-introspection — adapter rejection reasons classified as they are (invalid_event/recorder_error no longer miscounted as sink drops), merged capability declarations re-frozen and surfaced via status(); JSONL reading tolerates UTF-8 BOM; performance-budget gate (test/perf.test.mjs) added to npm test; compat gate now drives a real-host session-capture smoke (DSH baseline moved to 0.2.1-rc.2).
• Future iterations follow DSH-TELEMETRY-开发计划.md.

License

MIT