AI Cost Audit (token-cost-audit-en)
Turn "why did the AI bill jump again" into data you can take into a meeting: where the money went, what a task actually costs, what switching would save, whether self-hosting pays off, and which lever to pull first.
When to use
Trigger:
- "AI costs are too high", "audit our token spend", "how much did we spend on API calls"
- "how do I analyze OpenRouter / LiteLLM / Helicone usage", "Claude Code billing"
- "how much would switching to DeepSeek / GLM / a small model save, and what breaks"
- "is self-hosting worth it", "at what volume should we buy GPUs"
- Quarterly budget reviews, explaining AI ROI to leadership
Do not trigger:
- Just wanting a vendor pricing page or a leaderboard — that is news, not an audit
- Benchmarking model quality (running MMLU, SWE-bench and friends) — this skill does money and volume only
- Optimizing non-AI cloud spend (EC2, object storage, egress)
- Formal accounting treatment (capitalization, amortization policy)
Three things to settle first, or the numbers will be wrong
- Listed spend is not actual cost. Cache hits, batch discounts, free credits, flat subscriptions and grants all move the bill away from "tokens × unit price". Two public reference points: on OpenRouter, Chinese models account for roughly 65% of token volume but only 29% of listed spend — DeepSeek alone is about 32% of volume against 6% of spend. Reading a price table instead of your own bill produces conclusions that are off by half.
- Unit price is not comparable across models — cost per completed task is. A cheap model may emit longer output, retry more, and stuff more context, easily doubling real cost. An expensive model that gets it right in one pass can be cheaper. So the third column is always cost per task, never dollars per million tokens.
- Part of the saving turns into review labor. Drop a capability tier and rework plus review time goes up. Any savings figure must carry a "share requiring human review" column, or engineering will reject the plan in the first meeting.
Steps
0. Get the data (10 minutes, decides everything downstream)
Three sources, pick by availability, prefer the first:
| Source | What you get | How |
|---|---|---|
| Gateway logs (best) | per-call model, tokens, latency, caller, task tag | LiteLLM / Helicone / self-built gateway export to CSV or JSONL; without task tags, use the app name from metadata |
| Vendor billing | daily and per-model volume and amount | Anthropic / OpenAI / cloud console billing export |
| OpenRouter activity | per-model, per-key usage and spend | Settings → Activity → Export, gives one record per call |
Minimum column set (fill what is missing; if too much is missing, add instrumentation before optimizing):
timestamp,model,team,project,task_type,input_tokens,output_tokens,cache_read_tokens,cost_usd,tasks
2026-11-01T10:00:00Z,claude-haiku-5.5,platform,checkout,code_review,12000,1800,9000,0.0041,1
If cost_usd exists, use it — that is the accurate number. If not, the script estimates from the price table and the report says so.
tasks is the business volume for that row (one task, one PR, one session). Without it you can only reach dollars per million tokens.
Checkpoint: at least 14 days covering 80%+ of spend. One week gets skewed by release cycles.
1. Run the audit (one command)
python3 scripts/cost_audit.py --input usage.csv --prices prices.json --group-by team,task_type,model
Six sections out: totals (tokens / cost / blended price / cache hit rate), the four-dimension split, top spenders, unit economics, anomalies (a single row 10x the mean, cache hit rate below 20%, the same task spread across models), and any models missing a price.
Checkpoint: totals within 5% of the finance bill. If not, reconcile basis first (timezone, tax, credits).
2. Split the cost: find the actual 20%
Read three tables, not the total:
- By team — who burns it, and does that match their output
- By task type — 60–80% of spend usually sits in one or two task types (long-context code review, agent loops, bulk summarization)
- By model — is anyone using a flagship model to classify, tag or format
Checkpoint: you can state the top three spenders, their shares, and who owns each.
3. Unit economics (the only basis business will discuss)
Use the four formulas in references/unit-economics.md: cost per task, per pull request, per active user, per thousand lines of code.
The one-line conclusion must read like: "code review costs $0.42 per PR, and 71% of that is stuffing the whole repository into context" — not "we spent $3,800 this month".
Checkpoint: one unit metric that can be compared month over month and has a target.
4. Model-switch simulation
python3 scripts/cost_audit.py --input usage.csv --prices prices.json --switch-to glm-5.3-flash,deepseek-v4-flash
Output is saving, saving percentage and price drop per candidate. Then add two columns by hand — the script cannot supply them:
| Add | How to estimate |
|---|---|
| Capability gap | run 30–50 real tasks through both models, record first-pass success rate |
| Review delta | every 10 points of first-pass success lost means about 10% of output needs rework; multiply by hourly labor cost |
Net saving = model saving − rework labor − one-off migration cost. The third term is the one everyone forgets.
Checkpoint: every candidate has one sentence: "saves X%, needs Y% more review, net positive or negative".
5. Self-hosting TCO and break-even
python3 scripts/cost_audit.py --input usage.csv --self-host gpu_hour=2.4,gpus=8,utilization=0.55,ops_month=3000
The math: monthly fixed cost ÷ (current blended unit price − self-hosted variable unit price) = break-even monthly token volume. Below that volume, self-hosting is always more expensive.
Do not forget: idle capacity (real utilization is 40–60%), model generations falling behind, on-call staffing, and the fact that you will end up running a self-hosted primary plus an API fallback. The full checklist is in references/unit-economics.md.
Checkpoint: state "current volume is N times the break-even", not "self-hosting is cheaper".
6. Optimization list, sorted by ROI — do the first three
| Lever | Typical saving | Cost of doing it | When not to |
|---|---|---|---|
| Slim the context (retrieve relevant files, stop dumping the repo) | 30–60% | needs retrieval | task genuinely needs global understanding |
| Cache hits (stable prompt prefix + enable prompt caching) | 20–50% | prompt restructure | prompts are different every call |
| Model cascade (small model first, escalate on low confidence) | 25–45% | one extra hop of latency | latency-sensitive paths |
| Batch / offline queue (batch discount) | 40–50% | not real-time | interactive products |
| Retry governance (backoff + failure classification) | 5–20% | client changes | failure rate already low |
| Subscription instead of metered | varies per user | admin overhead | volatile usage |
Checkpoint: every row has a monthly dollar figure and an owner. No orphan items.
7. Report and three plans
Report order: one-page conclusion → totals → three split tables → unit economics → switch comparison → self-host verdict → optimization list → appendix (basis and data sources).
Give decision-makers three options, never one:
- Conservative — context slimming, caching, retry governance only; no model change. Lowest risk, usually 25–40%.
- Balanced — conservative plus moving 2–3 low-risk task types (classify, summarize, format) to smaller models. 40–60%.
- Aggressive — balanced plus moving primary coding and reasoning to open or domestic models, and evaluating self-hosting. 60%+, requires A/B and rollback.
Staged migration: move by team or task type, never more than 20% of traffic at once → watch first-pass success, review time and unit cost weekly → expand only after two clean weeks. The rollback switch must exist on day one.
Five traps
| Symptom | Cause | Fix |
|---|---|---|
| Your number is far below the bill | missing cache writes, image/audio billing, failed retries still billed | calibrate against the bill, not tokens × price |
| Switched to a cheap model, total cost barely moved | longer output, more retries, fatter context | compare cost per task, not unit price |
| Cannot split by team | logs carry no caller identity | add team/key tags at the gateway, then wait two weeks |
| Self-hosting looks very cheap | computed at 100% utilization | recompute at real utilization (40–60%) |
| Engineering rejects the plan | only savings were reported, not rework | every plan carries a review-delta column |
What done looks like
- Totals within 5% of the finance bill, with the basis written down (tax in/out, credits in/out)
- Top three spenders named with shares
- At least one unit metric comparable month over month
- Every switch candidate has saving / extra review / net result
- Self-host verdict is a break-even multiple, not an adjective
- All three plans carry a rollback path and an owner
Hard constraints
- Calibrate against the real bill; never present a public price table as a conclusion. Every price table carries a date and is re-checked after 30 days
- Always convert to cost per completed business task before comparing models
- No savings figure without the review delta beside it
- Never share a customer's usage detail or bill; redact client and project names before the report leaves the team
- Do not promise a specific saving percentage — give a range and its preconditions
