Token Saver (token-slim)
- Author: Evan Song · github.com/Songhonglei
- Repository: https://github.com/Songhonglei/better-agent-skills
- License: MIT
Helps AI agents and their users reduce token consumption by auditing workspace
files, identifying bloat, and guiding targeted cleanup.
Core Concept
Every project-context file (memory index, agent config, heartbeat, etc.) is
injected into every single session. Keeping them lean = direct,
compounding savings. The goal is a three-tier memory layout:
memory index ← always loaded, < 150 lines — pointers + essentials only
memoryres/*.md ← loaded on demand — reference tables, specs, backlogs
memory/YYYY-MM-DD.md ← raw daily log — rarely loaded
For full strategy details: see references/strategies.md
Mode Routing
After matching a mode, first read the corresponding reference file, then
follow that flow. Do not rely on memory.
| User intent | Mode | Must read first |
|---|---|---|
| First-time setup / "save tokens" | Mode A | references/mode-a-onboarding.md |
| Re-scan / "scan again" / "what else can I optimize" | Mode B | references/mode-b-rescan.md |
| Toggle brutal mode | Brutal Mode | see section below |
Execution Modes
Confirm mode (default)
Show each proposed change, wait for user approval before modifying.
Batch mode
Triggered when the user says "just do it" / "按你说的来" / "全部做". Execute all
changes without per-item confirmation; deliver a unified summary at the end.
Dry-run mode
Preview without modifying any files:
SKILL_DIR="$(dirname "$(readlink -f "$0")")/skills/token-slim" 2>/dev/null \
|| SKILL_DIR=./skills/token-slim
python3 "$SKILL_DIR/scripts/scan_workspace.py" --workspace . --dry-run
Or simply, from the workspace root:
python3 <skill-install-path>/scripts/scan_workspace.py --workspace . --dry-run
🔥 Brutal Mode — Toggle
Triggers:
- Enable: "enable brutal mode" / "开启狂暴模式" / "打开狂暴模式"
- Disable: "disable brutal mode" / "关闭狂暴模式" / "退出狂暴模式"
Enable flow:
1. Check the workspace agent config file for <!-- token-slim:token-habits-start --> anchor
2. If absent → append the full template (incl. brutal mode section — see references/mode-a-onboarding.md Step 8)
3. If present → replace the whole block with the full template
4. Reply: "🔥 Brutal mode enabled — concise outputs, no preamble."
Disable flow:
1. Find the block <!-- token-slim:token-habits-start --> … <!-- token-slim:token-habits-end --> in the agent config
2. Replace the brutal-mode section with just a header placeholder:
### 🔥 Brutal Mode (max output efficiency) <!-- Brutal mode: disabled -->
3. Reply: "✅ Brutal mode disabled — back to normal."
First-time use (after Mode A Step 8 completes):
Proactively offer:
> "🔥 Want to enable brutal mode? When on, the agent gives results only — no
> step-by-step narration. Saves tokens and reading time. Say 'enable brutal mode'
> to toggle (can be turned off anytime)."
Undo (rollback)
scan_workspace.py itself does not modify files and does not auto-backup.
Before any file modification the agent must:
1. Tell the user which files will be modified
2. Ask "Do you want me to back these up first?"
3. On confirmation, copy the soon-to-be-modified files to a timestamped dir:
<workspace>/.token-slim/undo-<YYYYMMDD-HHMMSS>/
4. Report the backup path so the user can restore later.
See references/mode-a-onboarding.md Step 3 for the canonical recipe.
tiktoken installation
token-slim uses tiktoken for accurate token counting (vs ~40-60% heuristic
error). Install attempts run in this order with automatic fallback:
python3 <skill-install-path>/scripts/install_tiktoken.py --workspace .
| Step | Source | Notes |
|---|---|---|
| 1 | PyPI official (pypi.org) | 2 retries |
| 2 | Tsinghua mirror | China-region acceleration |
| 3 | Aliyun mirror | China-region acceleration |
| 4 | Heuristic fallback | CJK/ASCII split, ~40-60% error |
BPE vocabulary cache is fetched automatically by tiktoken on first encode
from the OpenAI public blob (openaipublic.blob.core.windows.net) and stored
under <workspace>/.cache/tiktoken/ (CWD-anchored — survives $HOME wipes on
container/VM environments where only the working directory is persistent).
Check current state:
python3 <skill-install-path>/scripts/install_tiktoken.py --check --workspace .
scan_workspace.py auto-detects tiktoken at runtime; no manual setup needed
after installation.
Dependencies
Required:
- Python 3.8+ (python3 available on PATH)
Optional:
- tiktoken for precise token counting (install via the script above). Without
it, the scanner uses a CJK/ASCII heuristic with ~40-60% error.
Everything else uses the Python standard library.
What NOT to move
Never suggest moving:
- Behavioural rules, safety constraints, must/never guidelines
- Cron job configurations and failure protocols
- Active heartbeat tasks and retry queues
- Identity / persona core (soul/identity config content)
- Anything with ⚠️ or explicit "always loaded" markers
When in doubt, ask the user.
Key files
| File | Purpose |
|---|---|
scripts/scan_workspace.py |
Scanner — detects bloat, scores findings, supports --dry-run |
scripts/install_tiktoken.py |
tiktoken installer (PyPI → Tsinghua → Aliyun → heuristic) |
references/strategies.md |
Full strategy reference (7 strategies + scoring rubric) |
references/mode-a-onboarding.md |
Mode A: first-time setup workflow |
references/mode-b-rescan.md |
Mode B: on-demand re-scan workflow |
CLI reference
scan_workspace.py
python3 scan_workspace.py [--workspace <path>] [--json] [--dry-run]
| Flag | Default | Description |
|---|---|---|
--workspace |
$TOKEN_SLIM_WORKSPACE or current directory |
Workspace root to scan |
--json |
off | Output raw JSON instead of human-readable report |
--dry-run |
off | Preview mode: show what would change, modify nothing |
install_tiktoken.py
python3 install_tiktoken.py [--check] [--workspace <path>]
| Flag | Default | Description |
|---|---|---|
--check |
off | Only report install/cache status; do not install |
--workspace |
current directory | Cache goes to <workspace>/.cache/tiktoken. Honoured only if TIKTOKEN_CACHE_DIR is unset |
⚠️ Skills vs workspace-file tokens
Skills (SKILL.md): each skill injects only its name + description + location
per session — around 24 tokens. The full body is only loaded when the agent
explicitly reads it. Large SKILL.md files affect call latency, not per-session
baseline.
Workspace files (memory index, agent config, etc.): injected verbatim every
session — these are the real optimisation target.
