Quick Reference
| Task | Load |
|---|---|
| Creative writing (voice, dialogue, POV) | writing.md |
| Data processing (CSV, regex, encoding) | data.md |
| Academic/citations (APA, MLA, Chicago) | academic.md |
| Marketing copy (headlines, CTA, email) | copy.md |
| Translation/localization | localization.md |
Universal Text Rules
Encoding
- Always verify encoding first:
file -bi document.txt - Normalize line endings:
tr -d '\r' - Remove BOM if present:
sed -i '1s/^\xEF\xBB\xBF//'
Whitespace
- Collapse multiple spaces:
sed 's/[[:space:]]\+/ /g' - Trim leading/trailing:
sed 's/^[[:space:]]*//;s/[[:space:]]*$//'
Common Traps
- Smart quotes (
"") break parsers → normalize to" - Em/en dashes (
–—) break ASCII → normalize to- - Zero-width chars invisible but break comparisons → strip them
- String length ≠ byte length in UTF-8 (
"café"= 4 chars, 5 bytes)
Format Detection
# Detect encoding
file -I document.txt
# Detect line endings
cat -A document.txt | head -1
# ^M at end = Windows (CRLF)
# No ^M = Unix (LF)
# Detect delimiter (CSV/TSV)
head -1 file | tr -cd ',;\t|' | wc -c
Quick Transformations
| Task | Command |
|---|---|
| Lowercase | tr '[:upper:]' '[:lower:]' |
| Remove punctuation | tr -d '[:punct:]' |
| Count words | wc -w |
| Count unique lines | sort -u \| wc -l |
| Find duplicates | sort \| uniq -d |
| Extract emails | grep -oE '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' |
| Extract URLs | grep -oE 'https?://[^[:space:]<>"{}|\\^[]]+'` |
Before Processing Checklist
- [ ] Encoding verified (UTF-8?)
- [ ] Line endings normalized
- [ ] Delimiter identified (for structured text)
- [ ] Target format/style defined
- [ ] Edge cases considered (empty, Unicode, special chars)
