TokenLite 4
Reduce large tool outputs without hiding source code or losing access to originals.
Non-negotiable safety contract
- Never compress source code. Auto-detected code and
--format codereturn byte-for-byte original content withreason=protected:code. - Never compress content marked
--is-user-input. - Keep stack traces below 2,000 estimated tokens unchanged.
- Treat CCR as unencrypted local temporary storage. Do not cache secrets unless the user accepts that risk.
- Do not install global hooks automatically.
Agent workflow
- After receiving a large non-code tool output, run:
python "{{skillDir}}/scripts/tokenlite.py" sense --input OUTPUT_FILE --json
- Use the returned
contentfield. It is always present. - When
auto_compressed=true, retainccr_idfor selective retrieval. - Retrieve only what is needed:
python "{{skillDir}}/scripts/tokenlite.py" retrieve --id ID --grep "ERROR|timeout"
python "{{skillDir}}/scripts/tokenlite.py" retrieve --id ID --line 100-160
python "{{skillDir}}/scripts/tokenlite.py" retrieve --id ID --field results.0.error.detail
- If
reason=protected:code, use the originalcontentunchanged.
Prefer --input for multiline or large content; shell quoting can corrupt large --text arguments.
Commands
| Need | Command |
|---|---|
| Evaluate and optionally compress | tokenlite.py sense --input FILE --json |
| Explicit compression | tokenlite.py compress --input FILE --format auto |
| Decide without compression | tokenlite.py should --input FILE --json |
| Retrieve original or slice | tokenlite.py retrieve --id ID [--grep P\|--line N-M\|--field PATH] |
| Inspect CCR | tokenlite.py stats |
| Delete expired CCR entries | tokenlite.py cleanup |
| Audit recent sense calls | tokenlite.py audit --json |
| Reply budget guidance | tokenlite.py lean budget --user-msg "..." --json |
Run python "{{skillDir}}/scripts/tokenlite.py" --help for all flags.
JSON behavior
JSON array compression keeps visible records containing nested errors, failure states, rare categorical values, schema variants, and robust numeric outliers. It then adds deterministic representative samples. Full input remains retrievable through CCR.
Privacy and hooks
Telemetry is off by default. Enable it only with TOKENLITE_TELEMETRY=1; it remains local.
The optional SessionStart hook modifies user-level configuration. Preview and install it only after explicit user approval:
python "{{skillDir}}/scripts/hooks/install-hook.py" --dry-run
python "{{skillDir}}/scripts/hooks/install-hook.py"
python "{{skillDir}}/scripts/hooks/install-hook.py" --uninstall
See references/privacy-and-hooks.md before installing the hook. See references/algorithms.md for compression guarantees and references/ccr-design.md for local retrieval behavior.
