DSH Report Card

An independent, dynamic benchmark for DeepSeek Harness plugins. It installs
one plugin into a throwaway profile, boots the real harness with it, runs one
scripted turn against a local fixture endpoint, and reports what the plugin
costs: boot time, per-turn hook overhead, the tokens it adds to the assembled
prompt, the capability surface it contributes, and the static risk flags its
package carries.

Every number is a reading from a real harness process — wall clock, process CPU,
or the exact bytes of the request it sent — none of it estimated from source
code. Every card stores its samples, its digest, and the command that reproduces
it.

What already exists, and why this is not a duplicate

Searched 2026-11-10: the dsh-plugin-catalog 2026.1110.18259 (9,959 plugins)
and npm. The adjacent tools are all static scores or in-profile meters:

One line: existing scorecards read a plugin's files, and the closest dynamic
instruments price what is already in your profile — nothing installs a candidate
into a disposable profile and times its boot, a scripted turn, and the exact
request it produces; that is what this does, so it complements them.

What a card contains

See docs/example-card.md for a complete card of the
test fixture, and docs/example-card.svg for the
graphic version of the same card.

Every card is written three ways: JSON (the data), Markdown (the readable
report), and SVG (a self-contained 1200×900 graphic — grade badge, metric
tiles with noise floors, context breakdown, capability surface, risk flags).
The SVG opens in any browser, embeds in Markdown, and has no runtime
dependency. In a terminal the CLI prints a boxed, coloured card instead; a
redirected stream gets Markdown.

How it measures

┌──────────────── throwaway DSH_HOME ────────────────┐
subject package ► │ profile "rc-measured" ← shipped "headless" template │
│   subject plugin: an insert row, or the bundle list │
│   probe: inserted last, dumps the surface, exits    │
└────────────┬───────────────────────────┬────────────┘
│ surface runs              │ turn runs
▼                           ▼
spawn → probe dump → exit     spawn → session → turn → exit
boot wall/CPU + surface       assembly/step + exact request
DEEPSEEK_BASE_URL → local fixture

• Fresh state. Each report gets a new DSH_HOME, a new profile copied from
  the shipped headless template, and an empty working directory. Your real
  profile is never opened, so a measured plugin cannot disturb it.
• No model calls. DEEPSEEK_BASE_URL points at a loopback endpoint that
  speaks the Anthropic-style Messages SSE stream the shipped adapter parses. The
  reply is a fixed one-line text block, so the turn is deterministic — and the
  request is captured verbatim, which is the context-tax measurement.
• No background noise. The fixture overlay disables session-title generation
  and telemetry, and a discarded warm-up pair primes the file cache before any
  sample is recorded.
• Round-paired deltas. Each subject run is paired with the baseline run
  immediately before it; the delta is the median of those paired differences,
  which cancels slow machine drift. Grades charge only the part of a delta that
  clears the machine's measured noise (one standard error of the paired
  differences), so a loaded machine cannot turn a free plugin into a bad grade —
  and cannot hide real overhead either, because the raw numbers stay on the
  card next to their floor.
• Integrity. The card is hashed (sha256 over canonical JSON, digest
  excluded). Re-running the same subject compares against the previous card in
  the same directory and prints drift.

Install

Published on npm as dsh-report-card@1.2.0. It ships as a bundle with
metadata, locale/en.json, locale/zh.json and an icon. The plugin manager adds
the dependency and enables the bundle in one step; removing it undoes both:

plugin_manager install_bundle  target: "dsh-report-card@1.0.0"
plugin_manager remove_bundle   target: "dsh-report-card"

or from a terminal:

dsh plugin --profile web add dsh-report-card

For development from a checkout, pass a directory spec instead
(file:C:/path/to/dsh-report-card). install_bundle is the sanctioned path
because a bundle is composed through the profile's bundle list, not just
node_modules; the bundle inserts one entry, dsh-report-card, which
contributes one tool.

Use

From a session (the tool is model-facing):

report_card(pluginSpec: "dsh-trust-check@0.2.6", samples: 3)
report_cards_recent(limit: 10)     # the last cards on disk, including automatic ones

report_card measures on demand; report_cards_recent reads the card index so
you can see what the background pass produced.

From a terminal — the same code path, printable output:

node harness/cli.mjs ./test/fixtures/fake-plugin              # terminal card (TTY)
node harness/cli.mjs dsh-trust-check@0.2.6 --samples 3        # npm package
node harness/cli.mjs dsh-quality-score --json                 # raw card
node harness/cli.mjs --from report-cards/x/y.json --pretty    # re-render, no measurement

The terminal card is boxed and coloured when stdout is a TTY; a redirected
stream prints the Markdown card instead. --pretty / --markdown force either
form, --no-color drops ANSI, and --svg writes the graphic beside the
other outputs.

Cards are written to ./report-cards/@/.{json,md,svg} and
indexed in ./report-cards/index.json; set DSH_REPORT_CARD_DIR to move that.

The tool takes ~60–90 s for the default five samples: four harness boots per
sample round (surface × 2, turn × 2) plus one discarded warm-up round. Pass
samples: 3 for a quick look and samples: 8..10 when you need to resolve a
few milliseconds.

Automatic measurement

Since 1.1.0 the card is produced after a plugin is installed, without
anyone asking. The plugin listens to the plugin manager's
plugin-manager/operation event and watches the profile manifest, queues the
new package, and measures it in the background — the same throwaway profile,
three samples, one measurement at a time.

install a plugin
│  manager event (Plugins page, plugin_manager tool)
│  or profile package.json change (dsh plugin add)
▼
debounce 2.5 s → list installed third-party bundles → skip ones already carded
▼
queue → measure in a throwaway profile → save card under $DSH_HOME/report-cards
▼
report_cards_recent lists it

What it skips: shipped/optional/official bundles, bundles that failed to load,
its own package, and any name@version that already has a card. What it does
not do: measure plugins that were already installed before this version
started, unless autoBackfill is on.

The finished card is announced exactly once: the next prompt assembled after
the measurement carries a one-shot note with the grade and the graphic path, and
reading that note clears the flag — so a restart, a second session, or a later
turn cannot repeat the announcement. Automatic cards take roughly 45 s each
(three samples) and run serially; a manual report_card call queues behind a
running automatic card so the two never compete for the machine.

Config, in the profile patch (or the plugin's config in a preset):

- id: dsh-report-card
config:
autoMeasure: true      # false turns the background behavior off
autoSamples: 3         # 3..10; steadier numbers cost more time
autoBackfill: false    # true: card already-installed plugins at startup
autoReportsDir: null   # default: /report-cards

DSH_REPORT_CARD_AUTO=0 disables it for one process without touching config.

Reading a card

**Grade D** · graded boot +6.3 ms CPU · graded turn 0 ms · context +76 tok · risk D

• The headline shows the two graded deltas plus the exact context delta and
  the risk grade. Every raw number — wall boot, process CPU, turn, assembly,
  step, launch — is in the tables below with its samples and noise floor.
• Wall time is presented but never graded: it is hostage to whatever else the
  machine is doing. The boot grade uses noise-adjusted process CPU at the
  moment the plugin tree is ready; the turn grade uses the noise-adjusted
  assembly time, the phase plugin hooks run in.
• context is the median of the captured requests; it is exactly reproducible,
  since the prompt is a function of the profile.
• risk is disclosure, not a verdict: A none, B informational only,
  C medium flags, D high flags. A postinstall script is a high flag.
• The overall grade is the worst dimension. F means the card could not measure
  a running plugin.

Numbers are comparable only on the same machine identity (platform, CPU
count, dsh version) — the card prints it. A delta inside the printed noise floor
is noise; the card says so in its notes.

Reproduce and verify

Every card prints its own reproduce command. The harness's own repeatability
check runs one subject several times and reports the spread:

node harness/repeat.mjs ./test/fixtures/fake-plugin --runs 2 --samples 3

Measured while a harness session was actively running on the same machine (full
output in docs/acceptance.md): the context delta
repeated at 0.00%, the graded boot CPU and assembly deltas at 0.00%, and
the raw round-paired wall deltas within 1.9% (boot) and 2.4% (assembly). Earlier
in the same session, with a busier machine, raw wall deltas moved by up to 19%.
The card publishes its own jitter and noise floor for exactly this reason.

Tests:

node test/unit.test.mjs      # units, ~1 s
node test/auto.test.mjs     # install trigger and queue, ~1 s
node test/harness.test.mjs   # real boots against fixtures, ~80 s

test/harness.test.mjs covers the whole harness with fake plugins: a healthy
plugin (surface, context tax, risk flags, digest), a plugin that throws during
activation (must be a failed card), a plugin that calls process.exit (must be
a failed boot that still returns), and a second card that must show drift
against the first.

Publishing

The package is publish-ready: npm publish and the catalog PR are documented in
docs/publishing.md, including the exact
data/plugins/__.yml entry the awesome-dsh-plugin list expects, the
repository field the catalog's npm mapping depends on, and the tarball-install
check that was run against the packed artifact.

Safety and limits

• The subject executes. It runs in a disposable profile with a read-only
  file policy and a fresh DSH_HOME, as your user — that is enough to keep it
  away from your real profile, but it is not a security sandbox. Read the
  risk flags before measuring something you do not trust.
• No app.asar edits. The harness only spawns the shipped CLI and writes
  into a temp directory.
• Fail closed. A package the runtime refuses to install (for example a peer
  version gate) is reported as install-rejected or install-failed; the harness never grants a
  version exemption and never bypasses the gate. A plugin that crashes the host
  or hangs is bounded by a per-child timeout.
• The heuristic is not billing. Context tokens use four characters per
  token, the same heuristic the shipped token meter documents; CJK and JSON
  schemas are underpriced by it. Extension fields (dsh_session_log,
  dsh_plugin_packages) are measured separately and excluded.
• One turn, one profile shape. The card does not measure multi-turn
  behaviour, compaction, or client-side (browser) cost. Client slot
  contributions are not observable host-side; only host-side surface is
  reported.

Layout

License

MIT.