dsh-agent-observability
Community plugin for DeepSeek Harness. Event-grounded, privacy-first, eval-ready reliability analysis. No official affiliation or endorsement.
中文 · Demo report · Technical report · Study guide
Quick Start
Install Node.js ^22.19.0 || >=24, pnpm, and the tested Harness release. The plugin is published on npm.
npm install --global @deepseek-ai/dsh@0.2.0-rc.2 pnpm
dsh plugin --profile web add @beibeihk/dsh-agent-observability@0.1.0
Enable: dsh plugin add selects the bundle automatically. Confirm the agent-observability row:
dsh --profile web --dump-config
Run:
dsh web
Run your usual task in Web (your own model credentials are required), then submit /observe in that session. The command returns metrics and findings without a model turn. Reports also export automatically at durable turn completion and session flush.
View report: /observe prints its directory. Open report.html locally. Default location: $DSH_HOME/profiles/web/observability// (DSH_HOME defaults to ~/.dsh). Files: trace.json, trace.jsonl, summary.json, summary.csv, eval.jsonl, report.txt, report.html.
Alternatively, download the prebuilt .tgz from v0.1.0 and run dsh plugin --profile web add ./beibeihk-dsh-agent-observability-0.1.0.tgz. The registry package is byte-for-byte identical to that release artifact. Publication and registry installation verification.
Keyless, reproducible demo
git clone https://github.com/beibeihk/dsh-agent-observability.git
cd dsh-agent-observability
npm ci --ignore-scripts
npm run demo
The demo packs the built plugin, installs it with the real dsh plugin --profile headless add, and runs the real dsh --profile headless --patch ... with an offline deterministic adapter and tool. It uses an isolated project-local .demo-home, disables Harness telemetry for this experiment, and writes artifacts/demo/report.html, trace.jsonl, and summary.json. No paid API calls. Example details.
What it does
Reconstructs Session → Turn → Step → Model settlement → Tool call/result trajectories from committed session evidence. Computes reliability metrics, detects bounded observable patterns, cites exact event sequence numbers, and exports failure cases for later evaluation. Normalized schema.
Why
A console log cannot reliably distinguish retries from steps, a cancelled turn from an errored one, or a surface rewrite from a second tool result. The Harness session log records the facts the model context derives from; this plugin uses those facts without rewriting them.
Example Output
Session Reliability Report
Status: completed
Turns: 1 | Steps: 2 | Model requests (settled): 2
Tool calls: 1 | Successes: 1 | Failures: 0 | Cancelled: 0
Detected Issues:
- None detected
This excerpt comes from the offline Harness demo. Full output · Dashboard. The dashboard is static HTML with no scripts, external assets, or arbitrary overall score.
Failure Detectors
| ID | Observable pattern | Default |
|---|---|---|
| F01 | Call without terminal result in a closed, complete interval | Error; suppressed for cancellation/fork closure |
| F02 | Multiple terminal results in the same turn/step/call | Error; surface replacements excluded |
| F03 | Consecutive equivalent calls | 3 calls within 60 s |
| F04 | Same tool fails across consecutive steps | 3 failures |
| F05 | Excessive settled model attempts per step | More than 3 |
| F07 | Durable abnormal turn termination | Error/interruption; warnings for policy/token limits |
| F08 | Call-to-result latency above threshold | More than 30 s |
| F09 | High tool error ratio | More than 50%, at least 4 success/error results |
| F10 | Frequent logged request/context changes | At least 5; informational |
| T01 | Observed tool error | Informational; recovery can still complete |
F06 is deferred: model text and tool success do not objectively establish task progress. Findings detect evidence-backed runtime and trajectory patterns, not all agent failure causes or answer correctness. Definitions, examples and false-positive risks.
Architecture
The official dsh.bundle.patch inserts one Cordis plugin. It injects sessions and sessionQuery, subscribes to observe-only session/event, and replays seeds through asynchronous sessionQuery.observeSession() leases. A typed local ctx.dshObservability service exposes trace(), export(), flush() and a diagnostic. Optional ctx.commands contributes /observe as a human command. All listeners and services unwind with their Cordis fiber.
No loop modification, monkey patches, node_modules edits, waterfall interception, additional model tool schemas, or prompt injection. Architecture · Integration.
Privacy
Metadata-only by default. No telemetry is sent by this plugin. Capture requires captureContent: true; redaction remains enabled. Reasoning blocks and embedded assistant streams are always excluded. Exported content hashes cover redacted content; argument equality uses an ephemeral session-keyed HMAC. Opaque tool metadata is never copied. Privacy and threat limits.
The upstream Harness has its own telemetry and provider session-log behavior; installing this plugin does not change either. Our demo explicitly opts out of Harness telemetry. Capturing plaintext into Harness itself is outside this plugin's redaction layer.
Performance
npm run benchmark runs 130 labeled synthetic traces, a 10,000-event recorder microbenchmark and the published Session event-dispatch benchmark. On Node 24.12.0 / Windows x64, the recorder adds 9.62 μs/event in metadata mode; official Session.append plus observation adds 34.20 μs/event (+79.67% in that message-only microbenchmark). Content mode adds 27.64 μs/event in the latter. These figures exclude model latency and disk I/O and do not describe end-to-end task overhead. Heap/RSS deltas are noisy, GC-sensitive measurements. Raw results · Method and limitations.
The 130 synthetic traces produce 130 true-positive session/type label pairs, zero false positives and zero false negatives (precision/recall 1.0). They come from 13 designed templates with limited variation, not independent real-world tasks; this validates the specified rules, not general agent accuracy.
Memory is bounded by maxSessions (64), maxEvents (100,000 per retained session), and maxContentBytes (16 KiB per captured payload). Bounded history bootstrap and export queues report saturation. Writes are asynchronous batched snapshots at turn/flush boundaries, with atomic replacement of each file. There is no token-level write or fsync.
Compatibility
Tested against published DeepSeek Harness 0.2.0-rc.2, Cordis 4.0.4, and its current logical session format. Peers declare the exact tested prerelease; engines.dsh repeats it as metadata. No claim for other releases or 0.2.1-alpha.1. Compatibility evidence.
Development
npm ci --ignore-scripts
npm run validate
npm run test:coverage
npm run benchmark
npm run demo
npm pack
CI runs lint, typecheck, offline tests, build, synthetic benchmark and the packed official-launcher demo on Node 22.19+ and 24. Only published package exports are imported in production; deprecated synchronous session readers occur only in test files.
Limitations
Usage totals cover only assistant messages with reported counters; usage_coverage exposes coverage. Settled request counts omit attempts lost before durable settlement. No inferred context utilization, task-progress score, network trace, hidden reasoning, full PTC subcall model, or real-world accuracy claim. Fork metrics count child-owned work; inherited evidence remains explicitly marked. Config reload is tested as dispose/remount/replay; a filesystem-driven source-HMR stress matrix is not claimed.
Files are individually atomic, not a multi-file transaction. A hard crash can lose the not-yet-exported interval; previous completed JSONL snapshots remain parseable. No automatic disk retention deletion: remove old hashed session directories when needed. Retention and recovery.
Roadmap
Versioned exporters and detector functions are the extension seam for Agent Reliability Observatory adapters, OpenTelemetry, benchmark adapters and opt-in judges. V1 keeps deterministic rules and local files; it does not implement remote telemetry or judge-based labels.
MIT · © 2026 Kun Huang. DeepSeek Harness is a trademark of DeepSeek; this community project uses the recommended DSH naming convention.
