skill-smith
Forge skills that make agent behaviour predictable — the same process every run, not the same output. One spine holds this whole skill together:
> Static critique proposes; the run adjudicates.
You cannot read whether a line is a no-op, whether a leading word is strong enough, or where premature completion strikes — these are verdicts relative to the model's defaults, and only a run reveals the defaults. So every design judgement made in Steps 0–3 is a hypothesis, and Step 4's harness is the instrument that convicts or acquits it.
What's in this package
skill-smith is self-contained. Its two pillars are vendored, not fetched:
- Vocabulary & design judgement — every bold term above and below is defined in
references/glossary.md. Consult it when making a design call (invocation, hierarchy, splitting, pruning). Do not restate its definitions in skills you write: within this package it is the one source of truth. - Harness & mechanics — test-run orchestration (
scripts/run_eval.py,scripts/run_loop.py), grading (agents/grader.md), benchmarking (scripts/aggregate_benchmark.py), human review (eval-viewer/generate_review.py,assets/eval_review.html), JSON schemas (references/schemas.md), and packaging (scripts/package_skill.py) all live in this folder. Reach for this machinery in Steps 4 and 6; never rebuild it ad hoc. - Platform adapters — this SKILL.md follows the open AgentSkills spec and loads on any compliant platform (Claude Code, Codex CLI, OpenClaw, and others); harness capabilities vary. On any platform other than Claude Code, read
references/platforms.mdbefore Step 4 to pick the right instrument tier.
Provenance: the glossary is condensed from Anthropic's writing-great-skills; the harness is vendored from Anthropic's skill-creator (see LICENSE.txt). These are pinned copies — syncing upstream improvements is a deliberate maintenance act, not automatic.
Process
Work through the steps in order. Each ends on a completion criterion — do not leave a step until its criterion holds.
Step 0 — Choose invocation
Before any content exists, decide how the skill is reached. Model-invoked buys agent discoverability at permanent context load; user-invoked costs zero context but spends the human's cognitive load. Pick model-invocation only if the agent (or another skill) must reach it unprompted. If user-invoked skills are multiplying past memory, propose a router skill.
Done when: invocation is chosen, and the load being paid (context or cognitive) is named to the user in one sentence.
Step 1 — Capture intent and enumerate branches
Understand what the skill must do. If the current conversation already contains the workflow ("turn this into a skill"), mine it first — tools used, step sequence, corrections the user made, input/output formats — then have the user confirm the gaps. Interview for edge cases, input/output formats, example files, success criteria, dependencies.
Then enumerate the branches: every distinct way this skill will be invoked. Branches are not decoration — they are the disclosure test in Step 2 and the trigger list in the description.
Done when: branches are listed exhaustively and confirmed by the user, and the leading words the user actually says when they want this skill are identified.
Step 2 — Draft on the ladder
Write the SKILL.md. Place every piece of content on the information hierarchy:
- Steps (if the skill has them) in-file, primary — each ending on a completion criterion that is checkable (agent can tell done from not-done — resists premature completion) and, where it matters, exhaustive ("every X accounted for" — sets the legwork). These two axes are independent; demand binds flat reference too.
- Reference every branch needs: in-file, below the steps, co-located (a concept's definition, rules, and caveats under one heading).
- Reference only some branches reach: disclose behind a context pointer whose wording — not target — decides when the agent reaches it.
Anchor recurring behaviour with leading words — pretrained concepts repeated as tokens, never as sentences. Reach for an existing word before coining one.
Write the description as the top-level context pointer: front-load the skill's leading word, one trigger per branch (synonyms of one branch are duplication), plus a reach clause if other skills must invoke it. Bias toward firing — agents under-trigger skills — but get the reach from covering real branches, not from piling on synonyms.
Structural conventions (folder anatomy, scripts/ vs references/ vs assets/, frontmatter fields, imperative voice, example formatting): follow skill-creator's Skill Writing Guide. Explain why behind instructions rather than stacking bare MUSTs — ALL-CAPS is a yellow flag that reasoning is missing.
Done when: every step ends on a checkable criterion; every disclosed file is named from SKILL.md with the condition for reading it; no "when to use" text lives in the body (it belongs to the description).
Step 3 — Prune: propose the hypotheses
Audit the draft line by line before any run. This pass nominates suspects; it cannot convict.
- No-op test, sentence by sentence: does this change behaviour versus the model's default? Failures get deleted whole, not trimmed.
- Duplication: each meaning in one place. A triad restated at three sites is a passage begging to collapse into one leading word.
- Relevance: does the line still bear on what the skill does?
- Sprawl: even all-live, all-unique content can be too long — push reference down the ladder, or split by branch or sequence.
Done when: every surviving line either changes behaviour or is load-bearing reference, and each suspected-but-kept line is flagged as a hypothesis for Step 4 to test.
Step 4 — Run: the adjudication
Now point the instrument at the hypotheses. Use skill-creator's harness — test prompts in evals/evals.json, with-skill and baseline runs spawned together, grading, generate_review.py for human review. Get outputs in front of the user before forming your own conclusions.
Read the mapping right: the harness is not generic QA — each measurement adjudicates a specific design judgement.
| Instrument | Verdict it delivers |
|---|---|
| with-skill vs baseline delta | Is the skill as a whole a no-op? |
| Transcript reading (not just outputs) | Where legwork thins; where premature completion strikes; which leading word never fired; wasted work the skill is causing |
| Same prompt, repeated runs | Process variance — the predictability measure itself |
| Trigger evals on the description | Invocation reliability, per branch |
Repeated similar helper-work across transcripts is a verdict too: bundle it once into scripts/.
No harness available (no subagents, no scriptable CLI — see references/platforms.md for your platform's tier)? Degrade honestly: run each test prompt yourself following the draft skill, one at a time, and read your own transcript with the same table. Weaker instrument, same verdicts.
Done when: every hypothesis flagged in Step 3 has a verdict backed by run evidence — a transcript excerpt, a delta, or a trigger rate — not by re-reading the draft.
Step 5 — Cure by failure mode
Fix what the run convicted, using the failure-mode decision tree — not generic rewriting:
- Premature completion → sharpen the completion criterion first (cheap, local). Only if it is irreducibly fuzzy and the rush was observed in a transcript: hide the post-completion steps by splitting the sequence — across a real context boundary, or hiding clears nothing.
- Thin legwork → raise the criterion's demand, or plant a stronger leading word.
- No-op convicted → delete, or replace a weak leading word with a stronger one (be thorough → relentless).
- Sprawl / buried steps → disclose reference down the ladder; split by branch.
- Mis-triggering → reword the description's pointer; sharpen before inlining.
- Overfitting to test cases → generalize: the skill must survive prompts the user never wrote. Fiddly example-specific patches are a smell; explain the why instead.
Then loop: back to Step 4 with a fresh iteration directory until the user is satisfied, feedback comes back empty, or progress stalls.
Done when: each fix names the failure mode it cures and the run evidence that demanded it.
Step 6 — Tune the trigger and ship
Offer description optimization (scripts/run_loop.py — realistic should/should-not-trigger queries, near-misses included, user signs off on the eval set). This automation needs a scriptable agent CLI (claude -p, or a compatible binary via the AGENT_CLI env var); where none exists, tune the description manually against the same query list — see references/platforms.md. Then package and deliver: a plain folder is the canonical artifact; generate a .skill with scripts/package_skill.py only when targeting Claude.
Done when: the final description's trigger rate is measured (or the user declines), and the packaged skill is in the user's hands.
The mirror test
Before shipping any skill, hold it to the standard this one lives by: it should read like documentation written for the agent, every line paying for its context, its own design choices explainable in the vocabulary it was built with. If skill-smith's output cannot survive skill-smith's own audit, it is not done.
