🦃 token-gobbler

> same gobbling, different invoices. watch the corp tokens get chomped.
>
> Using DSH is running model evaluations. Every session — every prompt, tool call and turn —
> is a measurement of the model that served it. The token gobbler is the audit of that: which
> model did the work, how it performed, and whether the bill is honest.

A DeepSeek Harness composition plugin that reads your DSH session
data, counts every token you fed the machine (fresh input, output, cache reads, cache writes),
attributes those tokens to the models that actually produced them, and prices the whole gobbling
against the models your corp could've billed you for — Copilot's Claude / Grok — next to the free
local models you're actually running.

It's a little turkey with a spreadsheet. The turkey is you. The spreadsheet is the invoice.

The activity modal, Daily tab — filter by home (all sources / this machine / an imported one),
open a day, open a session: stat cards on top, that session's context & throughput chart underneath
(Trend mode here — one line per metric, the raw steps faint behind it, the ✂ compaction boundary
marked), and the turn-by-turn outline of every prompt, tool call and compaction at the bottom.

What it does

Reads the trajectory file DSH keeps per session — ~/.dsh/sessions///session.jsonl.zstd, a
multi-frame zstd JSONL event log — and reconstructs, per session:

• Model attribution — which provider/model served each step, matched against the projection
  store's authoritative totals and split across the models actually used. Best available: exact
  per-turn usage chunks, else estimated by step count, else the profile default model.
• Token buckets — uncached input, output, cache read, cache write (thinking is a subdivision of
  output, never double-counted).
• Speed — decode tok/s (streamed output ÷ time from first chunk to the usage chunk) and prefill
  tok/s (new input ÷ TTFT), plus per-step thinking time.
• Per-tool cost — the actual tool-call argument payload (~chars/4), not the whole step context.
• Pricing — a rate-card table (corp vs local) with a WFH / corp cost split.

The hard parts (multi-frame zstd decompression, per-step timing from chunk/step timestamps, a two-layer
in-memory + disk parse cache with a ♻ Reprocess reset) live in lib/trajectory.ts and
lib/report.ts.

Every DSH session is a model evaluation

You don't have to set up a benchmark — using DSH is the benchmark. The harness routes each
turn to whichever model the profile picked and records the whole exchange: the prompt, the tools
the agent called, the streamed output, the timings, the retries, the compactions, the outcome.
Every turn is one data point; a day of DSH work is an evaluation suite you already ran.

Token Gobbler is the audit of that suite, and it is deliberately model-agnostic: it reads the
same trajectory whatever produced it, so a local Qwen on llama.cpp, a DeepSeek API model and the
corp's Claude are all measured by the same yardstick.

• Which model did the work — per-step model attribution, including the step a session switches
  models mid-run (the ⇄ marker on the chart).
• How it performed — decode and prefill tok/s, TTFT, step wall-time, output per step, and the
  context growth the ✂ compaction boundaries reset.
• What it cost — tokens by bucket, priced against the corp rate card and the local one.
• How each turn went — its outcome badge (completed / aborted / error / max tokens / …), plus
  the retries, approvals, tool calls and compactions behind it.

Read it at three levels: the Runs tab compares a model's own runs like-for-like, the
Compare tab puts 2–4 runs of different models on one chart (resampled onto a shared progress
or runtime axis), and the invoice says whether the performance you paid for is the performance you
got. 🔌 Imported sources folds other DSH homes — other boxes, other OSes — into every number,
so the audit covers every model you run through DSH, anywhere. When you compare a local Qwen
against the corp's Claude, you're not just saving money; you're running a live benchmark and
keeping the receipts.

Install

npx @deepseek-ai/dsh plugin --profile web add github:Psynosaur/dsh-token-gobbler

Then restart the harness. Or add "token-gobbler": "link:/path/to/token-gobbler" to the profile's
dependencies and "token-gobbler" to its dsh.profile.bundles, then pnpm install.

Usage

node bin/token-gobbler.js          # pretty report
node bin/token-gobbler.js --json   # machine-readable
node bin/token-gobbler.js --days N # time window

Web dashboard

Adds a Token Gobbler section to the DSH settings modal: a stack of collapsible drawers — 💰
Cost, 🪙 Tokens, ⚡ Speed, 📊 Activity, 📈 Chart defaults, 🔌 Imported sources —
where each head carries the number that matters while the cards stay folded away. It holds the ↻
Refresh button plus a ♻ Reprocess button that clears the trajectory parse cache and re-parses
every historic file. The activity modal (the 🦃 button, bottom right) has tabs: Overview,
Cost, Performance, Runs (sessions grouped by the model that served them, one line per
metric over the runs, like-for-like — a model is only ever compared against itself), Compare
(pick 2–4 runs of any models out of a filterable list and read their per-step decode, prefill,
step time and running output on one shared axis — by run progress or by elapsed runtime, so runs of
different lengths line up), Daily (a GitHub-style calendar heatmap of token usage per day —
hover a day for its combined stats, click to filter the per-session table to it), Llama Metrics
(live llama.cpp server polling) and Settings (the rate cards).

Charts

Every chart is drawn by the plugin's own canvas engine (client/graph.ts +
client/graph-canvas.tsx) — no chart library is bundled, and none is injected into the host page.
Each chart opens in one of six plot modes, switched per chart above the plot: Lines, Both
(lines plus a dot per point), Dots (a scatter), Trend (a rolling median/mean/EMA through the
dots, the raw points left faint behind it), Bars (one bar per point, grown from the axis floor —
for discrete values like a day's cost) and Heat (one row per metric, one column per step, shaded
by value). Legend chips hide a whole group (a compaction window, a home), metric chips hide one
metric across every group, and the value axes rescale to whatever is left; hiding a metric is also how
you read two scales that differ by orders of magnitude. Which mode and which chips a chart opens with
are remembered per chart, and what they open with by default — plus how the lines themselves are
drawn (curved through the points or straight between them, how round the curve is, and whether Trend
mode's own rolling line is curved too) and how Trend and Heat behave — is set once in Settings ›
Token Gobbler › 📈 Chart defaults. Smoothing is monotone cubic (Fritsch–Carlson): the curve passes
through every step exactly and never overshoots between two of them, so a smoothed chart cannot show a
spike the data does not have.

Imported sources — other machines & other OSes

Your other boxes count too. Settings › Token Gobbler › 🔌 Imported sources registers ANOTHER
machine's DSH home and folds its sessions into every number — the Windows install on a mounted NTFS
volume (/media///Users//.dsh), a macOS home on /Volumes, a nightly rsync of a
laptop's ~/.dsh, or just a copy of its sessions/ folder.

• Add — type the path (or hit 🔍 Scan for DSH homes, which probes /mnt, /media/,
  /run/media/, /Volumes and your home for the shapes a DSH home takes). The layout is
  detected (a full home, the sessions/ folder, a projection store, or a bare tree of trajectories)
  and the OS is read from the sessions' own cwds — a Windows session says D:\models\… no matter
  where the drive is mounted today.
• Marked, everywhere — every imported session carries its home's badge (🪟/🍎/🐧 + name) in the
  session tables and drawers, and the activity modal gets a chip row to filter every tab down to one
  home.
• Manage — ↻ Resync re-reads that home only (its cached parses are dropped; the rest of the
  dashboard keeps its warm cache), ⏸ Pause keeps it registered but out of the totals, ✕ Remove
  forgets the import (the files on that machine are never touched).
• Read-only and honest — the registry is /token-gobbler/sources.json; nothing is
  ever written into an imported home. A session id that exists in two homes is counted once (this
  machine wins), and a home with trajectories but no projection store still reports: its rows are
  rebuilt from the trajectory.

Turn outline & the event timeline

Every session drawer (Daily, Combined, Performance, Tokens, Sessions, Cost …) opens with a Turn
outline & events section: one card per turn with its outcome badge (completed / in progress /
aborted / blocked / error / max tokens / interrupted), the prompt, the response (or the tools a
tool-only step called), the wall time and the step count — then that turn's compact event chips.
Events that belong to no turn (the system prompt, the session title) sit on a session-level row.
The per-turn → step table repeats the outcome badge and the first four event icons on each Turn
header row.

Events come from the trajectory records the parser previously only tallied:

A session whose trajectory has no turn records (older sessions) still shows its turns, derived from
the step tree, without prompts.

Trajectory formats & shape snapshots

Token Gobbler reads DSH session trajectories from ~/.dsh/sessions///. DSH
has shipped two on-disk shapes, parsed by the same pass:

• v0 — session.jsonl.zstd: one TOP-LEVEL record per streaming chunk (assistant/chunk with
  chunk.type usage/finish/…, plus text-chunks / tool-call-chunks / reasoning-chunks).
  The per-step tokens live in the usage chunk and the serving model in
  chunk.finish.replayState.response.
• v3 — session.v3.jsonl.zstd (current): the stream is NESTED in
  assistant/message.data.stream and data.usage mirrors its usage chunk; finish no longer
  carries replayState (the model is on message.source), the system prompt moved from
  request/header to a system/message record, and the header gained isSeeded. A directory
  holding both files parses the v3 copy only.

Because a format change used to look identical to "the numbers went to zero", every parse also
records the SHAPE of what it read. Snapshot it and diff an upgrade against it:

npm run report:shape                                       # shape of what is on disk right now
node scripts/trajectory-shape.js --out docs/trajectory-shapes.md
node scripts/trajectory-shape.js --compare docs/trajectory-shapes.md
# exit 0 = the format did not move; exit 1 = a record type / field / value set / stream kind changed

The manifest is Markdown for reading and carries the machine-readable snapshot in a fenced block.
See lib/trajectory.ts (the "FORMAT VERSIONS" header) for the field-level differences.

Pricing

cost = (uncachedInput·input + output·output + cacheRead·cacheRead + cacheWrite·cacheWrite) / 1e6

Pricing is table-driven: a model prices under its own entry, family fallbacks resolve for unseen
self-hosted models, and the first table is seeded from the trajectory (known models prefilled with
their real rates). Every entry carries a kind — corp (billed, in the company bucket) or
local (the home lab, priced at its configured rates). Edit lib/pricing.ts, or the Pricing tab
(saved to ~/.dsh/token-gobbler/pricing.json), where you set each model's kind, its rates, and the
★ WFH reference model without touching code.

Build

npm install     # one-time: esbuild + typescript
npm run build   # build:lib (tsc lib/*.ts -> lib/*.js) + build:client (esbuild -> lib/client.js)
npm run typecheck
npm test        # builds lib/*.js, then runs the node --test suite

Edit the .ts sources, never the generated .js.

Layout

client/*.ts(x)     web dashboard (TypeScript, bundled by esbuild)
client/graph.ts    the canvas chart engine every chart is drawn by (dependency-free)
lib/index.ts       host: /token-gobbler/* routes
lib/trajectory.ts  zstd trajectory reader/parser (v0 + v3) + shape capture
lib/projcache.ts   projection-store reader
lib/sources.ts     imported DSH homes: registry, layout + OS detection, scanning
lib/pricing.ts     rate cards + cost math
lib/report.ts      aggregate + attribution + pricing
lib/client.js      GENERATED (esbuild); do not edit
bin/token-gobbler.js  CLI (pretty/--json/--days/--breakdown)
scripts/trajectory-shape.js  trajectory shape snapshot/diff CLI
docs/trajectory-shapes.md    the pinned shape manifest (`npm run report:shape`)
test/report.test.js   node --test suite (plus sources / trajectory / drawers / graph / settings /
llama-metrics)

MIT. Not affiliated with any turkey, corp, or token ledger. The gobbler does not eat
your tokens — it audits them.