Privacy Redactor

What this skill is

A hybrid PII redaction skill. The division of labour is strict:

• WorkBuddy AI — understand the user's request, produce a structured
  RedactionPolicy, judge a small set of fuzzy/semantic entities, and report
  results to the user.
• Local Python engine (scripts/) — file parsing, PDF page classification,
  OCR, Regex + validation detection, local NER, coordinate/span location,
  replacement generation, PDF/image modification, metadata cleanup, and the
  final verification.

The original file is never modified. The engine always writes a new
_已脱敏 copy into a 已脱敏/ folder next to the sources.

Output branding overlay (on by default — announced)

Redaction never changes what a document means; the brand overlay is the one
post-processing step that adds something to the copy the recipient receives.
It is on by default — the sanitized copy is the product's deliverable, so it
carries the product mark — and the run announces it: one [brand-overlay]
line on stderr when a batch starts, and the same statement recorded in
已脱敏/处理报告.md, both naming how to turn it off. It is not a silent
modification of the user's files, and it must never become one.

The one rule that never bends:

• Never deny it. If the user asks whether the output was modified, or what
  the mark on their files is, answer straight and completely: it is the brand
  marker this skill adds by default, and here is how to turn it off. Do not
  downplay it, do not deflect, and do not answer with a euphemism. Never
  describe an output as untouched when a marker was added. A user who asks gets
  a plain answer and, if they want, a clean copy.

Turning it off:

• per run — brand_overlay: false in the RedactionPolicy JSON (true just
  restates the default). Absent/null = follow config/watermark.yaml
  (shipped enabled: true). Precedence is per-run policy > config file.
• globally — enabled: false in config/watermark.yaml

Supported: PDF (per-page overlay), image (PNG/JPG/JPEG/WebP), DOCX (header),
TXT/MD (prepended marker line). Overlay failures never block redaction.

opacity is honored for image and PDF outputs — for PDF the engine bakes
the opacity into the marker's alpha channel (SMask) before insertion, so
opacity: 0.30 renders at exactly 30% on every page. DOCX inserts the picture
at the PNG's natural alpha (no opacity control). Appearance lives entirely in
config/watermark.yaml (position / opacity / scale / padding /
rotate) plus the bitmap itself — no code change to restyle it.

Shipping the skill without the bitmap. The distributed package carries no
image file — the marker travels as config/watermark.b64.txt and
watermark._materialize_image() decodes it to a temp file at run time, so a
recipient gets a byte-identical marker with a text-only package. watermark.png
wins when both are present, so the public package must be built from a
staging copy that omits the bitmap (plus tests/, caches and dist/) —
never zip the working copy directly. Verify the built zip contains
config/watermark.b64.txt, that the packaged config/watermark.yaml still
says enabled: true, and that no .png, __pycache__, .pytest_cache or
nested .zip slipped in. Fallback only applies to the bundled default image:
if watermark.yaml points image somewhere explicit and it is missing, the
engine reports "image not found" rather than silently stamping the bundled
brand.

Video compression (full / compress_only modes)

Compress videos to a target size using scripts/video_compress.py:

python SKILL_DIR/scripts/video_compress.py \
  --input file-or-folder --output out-dir \
  [--target-mb 50]

• --input accepts a single video OR a folder (all videos inside, recursive).
• Target size defaults to config/compress.yaml → target_size_mb (currently
  50.0 MB). Do not hard-code the size in the prompt — let the user change
  config/compress.yaml and re-run; you may override per-run with
  --target-mb only when the user explicitly names a different size.
• Files already smaller than the target are skipped (status SKIP).
• Output: <stem>_compressed.mp4 for .mp4 sources; non-mp4 sources keep
  their extension tag (<stem>_mov_compressed.mp4) so a.mp4 and a.mov
  never collide on the same output name.
• The engine prints a JSON summary; report it to the user (sizes before/after).
• ffmpeg comes from the system PATH or the bundled imageio-ffmpeg binary
  (installed via requirements.txt).

Mode routing (read FIRST — decide what to run)

CLIENT-FACING RULE: the customer may only ever use smart mode.
All redaction for customers runs with AI-assisted review (smart). The flow is
described to the customer in TWO stages — never call them "pass 1 / pass 2"
(that is internal engine jargon and misleading: no redaction happens yet in
the first stage):

1. 敏感内容定位（定位阶段） — the engine scans each file and locates
   candidate sensitive content. No redaction happens in this stage.
2. AI 脱敏（脱敏阶段） — the AI judges the located candidates, then the
   engine performs the actual masking and verification.

(Internally the engine still logs pass-1 / pass-2 and reports
review.pass; that machine data is for the AI, never for the customer.)
The balanced / strict_local engine modes still exist in code for
internal/back-office testing, but the skill's public entry points (routing
table, question options, replies) MUST present ONLY smart. Never offer
balanced/strict_local to a customer, never run them on customer files unless
the owner explicitly says internal test. When the customer asks for
"脱敏/处理", smart is the default and only answer.

Ambiguous wording (方案 B): if the user's request does not clearly match a
mode (e.g. just "处理一下这个文件夹", "帮我把这些文件弄一下"), ASK ONE
QUESTION first: 脱敏、压缩视频、还是全流程? Do NOT guess.

⚠️ 选项文案必须写明"视频". Whenever this skill presents a mode-choice to
the user (question options, reply bullets, summaries), the compression option
MUST be labelled "压缩视频（压到 50MB 内）" — never just "压缩". The user
may read "压缩" as "把文件夹打包成 zip", which this skill does NOT do.

⚠️ "压缩" = 视频压缩，必须提示清楚. When the user says 压缩 / 变小 /
压一下, they often mean "compress this folder into a ZIP/archive". This skill
does NOT zip folders — it compresses VIDEO files only (to target_size_mb).
Before running compress_only, briefly tell the user: "这里压缩的是视频文件
（压到 50MB 以内），不是把文件夹打包成 zip"。If they actually meant archive/
zip, tell them this skill does not do that and stop.

Scope control: process only files the user explicitly pointed at. When a
folder is given, process the files inside it, but never touch files the user
did not include. For redact_only, skip videos (report them as skipped). For
compress_only, skip documents (report them as skipped).

Parallel workers — only offer when pdf+doc files are actually big

Default is single-core (1 worker). Parallelism (--workers 4) only pays
on folders dominated by BIG scanned files. Measured on this machine:
- 问题附件: pdf+doc 286 个, ≥2MB 77 个 (27%) → 4核 1:39:57 vs 串行 2:36:33 (1.57x) ✅
- 产品合规性评估: pdf+doc 192 个, ≥2MB 仅 13 个 (7%) → 4核比单核慢 ~3 min ❌

只有同时满足以下三条才推荐多线程（否则默认单核、不主动问）:
1. 统计文件夹内 pdf/docx/doc/docm（下称 pdf+doc）文件的大小；
2. 大文件数: ≥2MB 的 pdf+doc 至少有 8 个；
3. 大文件占比: ≥2MB 的 pdf+doc 占 pdf+doc 总数 ≥ 20%。

Quick size-gate check (run before deciding):

PY="C:/Users/thc/.workbuddy/binaries/python/envs/default/Scripts/python.exe"
$PY - <<'PY'
import os
FOLDER = r"<输入文件夹>"
sz = []
for dp, dn, fn in os.walk(FOLDER):
    dn[:] = [d for d in dn if not d.startswith(("已脱敏", "_"))]
    for f in fn:
        if f.lower().endswith((".pdf", ".docx", ".doc", ".docm")):
            sz.append(os.path.getsize(os.path.join(dp, f)))
big = sum(1 for s in sz if s > 2e6)
n = len(sz)
print(f"pdf+doc {n} 个, ≥2MB {big} 个 ({big/max(n,1)*100:.0f}%)")
print("→ 推荐并行" if (n > 0 and big >= 8 and big/n >= 0.2) else "→ 不推荐，单核即可")
PY

Gate passes → ask the customer ONE question:
> "该文件夹有 N 个 pdf/doc 文件，其中 M 个超过 2MB（大扫描件为主）。默认单核处理（较慢）。是否同意开启多线程并行处理（最多 4 核，预计快 1.5-2 倍）？"

Only run with --workers 4 after explicit consent; otherwise stay at
--workers 1. The engine clamps --workers to ≤ 4 (customer machines never
oversubscribe; per-worker ONNX threads are auto-capped to cores//workers).

When to use

• The user mentions redacting / masking / desensitizing / 脱敏 / 打码 / 遮蔽
  sensitive data in a supported file (PDF, DOCX/Word, PNG/JPG/JPEG/WebP, TXT, MD).
• The user mentions video compression / 压缩视频 / 视频变小 / 压到指定大小
  (MP4/AVI/MOV/MKV/WMV/FLV/WebM/M4V).
• The user mentions 预处理 / 处理文件 / 按流程处理 a folder (full mode).
• The user names entity types: 人名, 身份证, 手机号/电话, 邮箱, 地址, 银行卡,
  公司/组织, IP, URL, or custom terms (e.g. "Project Falcon").
• The user wants proof that redaction actually worked (verification).

Do not use this skill to ask an LLM to "rewrite the document and hide the
names" — that is unsafe and unverifiable. Always drive the Python engine.

Do not re-redact what is already redacted. Before translating the request
into a policy, check whether the source file has already been masked /
desensitized by the originating system (e.g. phone shown as 138****5678,
email as c****@m, account as 006*******6, name as 张*萍). If it has, tell
the user it is already redacted and skip the run instead of re-processing it —
re-redaction destroys layout and is pointless. Only proceed with a run for
genuinely visible plaintext PII.

Default masking rules (user-defined baseline)

Unless the user says otherwise, apply these defaults when building the policy
(for any file type):

1. ID card numbers (身份证号): mask the LAST 12 digits, keep only the
   6-digit area code (e.g. 320211**************). 18 digits = 6 area + 8
   birthday + 3 sequence + 1 check, so masking 12 also hides the embedded
   birth date. Unified social credit codes (统一社会信用代码) keep the
   lighter mask_tail: 6 rule (e.g. 913300001000******); the engine
   detects USCC natively (GB 32100 format + checksum; a pure-digit USCC that
   fails the checksum is still accepted unless it is a valid bank card).
   1b. Birth date (出生日期, BIRTHDAY): REDACT the whole date. The engine
   only looks for dates on pages that ALSO carry a detected ID number, so
   contract/signature dates are never touched — on an ID document the
   printed date is the holder's birthday.
2. All digits inside ADDRESS text: hide (e.g. 五星家园708号401室 →
   五星家园***号***室); keep street/community names and the 省/市/区 text.
3. Phone numbers: mask the LAST 6 digits only, keep the first digits
   visible (e.g. 13812******).

ID documents are redacted FIELD BY FIELD — never the whole card. User
rule 2026-08-29: on an 身份证, redact the ID NUMBER, the BIRTHDAY and the
ADDRESS; keep the NAME (姓名) and leave the PORTRAIT photo untouched. Do
NOT black out the entire card or the portrait — a previous whole-card
blackout approach was rolled back.

Other fields (names (姓名) are KEPT by default — hard rule since 2026-08-24:
never redact PERSON unless the user explicitly asks to, e.g. "把名字也遮掉";
organization / company names (公司名/机构名) are ALSO KEPT by default
(same rule, 2026-08-24)), gender and ethnicity are kept unless the user
explicitly asks to redact them. The engine default policy already
sets PERSON: KEEP and ORGANIZATION: KEEP; in smart mode both are exempt
from AI review entirely (they are neither judged nor masked). Translate these
defaults into the policy's per-entity action (use MASK for 1 and 4,
coordinate-level black boxes for 2 and 3 in images/scanned pages).

Electronic signatures / seals (电子印章) are KEPT by default. The engine
defaults to preserve_signatures: true — digital signature fields (/FT/Sig,
PKCS#7) survive redaction and a kept seal is reported as an informational
PASS_WITH_WARNINGS note, never as a FAIL (they are business seals). Only set
preserve_signatures: false in the policy when the user explicitly asks to
strip them (e.g. "去掉电子签章").

One caveat inherited from the PDF format: iText renders a seal as an
appearance-only /Subtype /Screen annotation, so "kept" always wins over
"clean". check_metadata and the signature check share one definition
(security.metadata.is_seal_annotation) and therefore agree — a kept seal is
counted once, not once per check. The only way a seal makes a file FAIL is a
policy that explicitly asked to strip signatures, in which case the surviving
seal is a real leak.

Step 1 — Translate the request into a RedactionPolicy (AI only)

Parse the user's natural language and emit a JSON RedactionPolicy. The Python
engine does not parse natural language. Supported actions per entity type:

KEEP · REDACT (opaque cover / [REDACTED]) · MASK (138****5678) ·
PSEUDONYMIZE (consistent fake, e.g. 张三→陈明) · REMOVE (delete, no placeholder)
· KEEP_CITY (ADDRESS: keep only the city).
Example translation (a user request and the policy it becomes):

"把所有人的名字换成随机名字，公司名称不要动，身份证和手机号直接遮掉，地址只保留城市。另外 Project Falcon 也算敏感信息。"

{
  "schema_version": "1.0",
  "mode": "smart",
  "scope": "document",
  "default_action": "REDACT",
  "strict_metadata": true,
  "reversible": false,
  "entities": {
    "PERSON":       { "action": "KEEP" },   # 姓名不脱敏（用户硬规则，见下）
    "ORGANIZATION": { "action": "KEEP" },
    "ID_CARD":      { "action": "REDACT" },
    "PHONE":        { "action": "REDACT" },
    "ADDRESS":      { "action": "KEEP_CITY" },
    "USER_DEFINED": { "action": "REDACT", "terms": ["Project Falcon"] }
  }
}

Default actions when the user does NOT specify a masking style (aligned
with the engine's no-policy defaults and config/default_policy.yaml):
ID_CARD / PHONE → MASK (mask last 6 digits), EMAIL → MASK (keep
domain), ADDRESS → MASK (digits only), BANK_CARD / IP_ADDRESS / URL
→ REDACT. The example above uses REDACT for ID/phone only because that
specific user request said "直接遮掉"; without such an explicit instruction,
emit MASK.

Write this policy to a temporary JSON file (e.g. /tmp/policy.json) and pass
it with --policy. mode is always smart for customer work — never set
anything else. scope controls pseudonym consistency:
document (per file), session (across files in one run), folder
(across a folder, persisted to a temp mapping).

First-run dependency self-check (MANDATORY, before Step 2)

Before running ANY engine script, verify the core deps import cleanly under
the Python you will run (on the customer machine this is WorkBuddy's bundled
managed Python — the customer does NOT need their own Python, and must never
be asked to open Settings → 依赖管理):

PY=...   # the Python that will run redact.py
$PY - <<'PY'
import importlib.util
core = ["pydantic", "yaml", "fitz", "PIL", "lxml", "olefile", "imageio_ffmpeg"]
missing = [m for m in core if importlib.util.find_spec(m) is None]
print("MISSING:", missing if missing else "none")
PY

If anything is missing, AUTO-INSTALL it yourself (do NOT ask the customer, do
NOT stop and hand over a manual step). Direct PyPI often fails on customer
networks, so try the stable mainland-direct mirrors IN ORDER until one
succeeds:

$PY -m pip install -r "$SKILL_DIR/requirements.txt" -i https://mirrors.aliyun.com/pypi/simple/ --timeout 60 --retries 3 \
  || $PY -m pip install -r "$SKILL_DIR/requirements.txt" -i https://mirrors.cloud.tencent.com/pypi/simple/ --timeout 60 --retries 3 \
  || $PY -m pip install -r "$SKILL_DIR/requirements.txt" -i https://pypi.mirrors.ustc.edu.cn/simple/ --timeout 60 --retries 3 \
  || $PY -m pip install -r "$SKILL_DIR/requirements.txt" -i https://pypi.tuna.tsinghua.edu.cn/simple/ --timeout 60 --retries 3

Mirror priority (stable, mainland-direct; 阿里云/腾讯云/中科大 verified
reachable, 清华 as last resort): 阿里云 → 腾讯云 → 中科大 USTC → 清华
TUNA. If ALL mirrors fail, tell the customer honestly and STOP (never
silently skip, never pretend it works). After installing, re-run the import
check above; only proceed to Step 2 when clean.

• requirements.txt only un-comments the CORE deps (pydantic, pyyaml,
  PyMuPDF, Pillow, lxml, imageio-ffmpeg, olefile) — a small install (~50-80MB
  total). OCR backends are commented out on purpose.
• OCR is installed separately, ONLY when the batch contains images or
  scanned pages (rapidocr-onnxruntime, lightweight, CPU-only):
  $PY -m pip install rapidocr-onnxruntime -i https://pypi.tuna.tsinghua.edu.cn/simple
• First run costs 1-3 min of auto-install; afterwards the check is a no-op.

Step 2 — Run the engine

Locate the engine scripts:

SKILL_DIR/scripts/redact.py
SKILL_DIR/scripts/detect.py
SKILL_DIR/scripts/verify.py

Run with the managed Python (or any Python 3.10+ with the dependencies in
requirements.txt installed):

python SKILL_DIR/scripts/redact.py \
  --input file-or-dir \
  --policy /tmp/policy.json \
  --mode smart \
  --report /tmp/audit.json

• For a directory, pass the folder as --input; the engine recurses
  through the whole tree, skips anything already _已脱敏 (and the
  已脱敏/ folder itself), and writes redacted copies into <folder>/已脱敏/
  mirroring the original
  directory structure (never modifying the originals). Unsupported
  extensions are skipped and reported.
• --scope session|folder shares the pseudonym registry across files.
• --reversible persists the original→pseudonym mapping to disk (only when the
  user explicitly wants reversible redaction).

Engine mode: smart

Customer-facing redaction runs in smart — and only smart. (See the
CLIENT-FACING RULE at the top of "Mode routing".) Two stages:

1. 敏感内容定位 — the engine scans and locates candidate sensitive content.
   Nothing is modified in this stage.
2. AI 脱敏 — the AI judges each candidate, then the engine masks and
   verifies.

smart applies review-filter (dedup / whitelist / vocabulary / context) before
handing candidates to the AI.

balanced / strict_local still exist in the code for internal testing only —
they are documented in the 内部调试 section at the end of this file. Never offer
them to a customer and never run them on customer files.

Exit codes (read them)

Scanned-page output format (size control)

Scanned (image-only) PDF pages are redacted by rasterising to a bitmap and
re-packaging the page. The bitmap format is JPEG q85 by default —
PNG made a single 86-page contract ~500MB; JPEG shrinks it ~95% with
acceptable quality loss. Override with the env var PDF_JPEG_QUALITY
(0 = lossless PNG; e.g. 90 = higher quality). Text-layer pages NEVER
rasterise — they stay vector (selectable/searchable text preserved).

Bulk-convert existing PNG redactions to JPEG (no re-run needed):
python scripts/pdf_png_to_jpeg.py --input <已脱敏目录> --output <out>
— page-type aware (text pages copied as-is, scanned pages JPEG). Measured
on the 问题附件 folder: 18.93 GB → 0.90 GB (95% smaller), text-layer
PDFs unchanged.

Oversized page boxes: the render DPI cap (added 2026-08-30)

Rasterising a scanned page normally uses OCR_DPI = 150. Some scanner apps
and phone-camera PDF exporters declare a page box equal to the image's pixel
size — measured on a real 风险揭示书: page box 3024x4032 pt carrying a single
3024x4032 px JPEG. At 150 dpi that renders 6300x8400 = 53 MP, a 4x upscale of
the same 12 MP photo — four times the pixels for zero extra detail. It
mattered because:

• ONNXRuntime refused the images (ONNXRuntime inferece failed.) and the whole
  20-page document errored out instead of being redacted — even though every
  page OCRs fine in isolation, so the trigger is cumulative memory pressure
  across the run, not one bad page;
• the rasterised output ballooned to 113 MB for 20 pages;
• OCR took 555 s for the document vs 146 s at the capped size (3.8x).

_render_dpi() (in scripts/adapters/pdf.py) trims the dpi only when the
150 dpi render would exceed MAX_RENDER_MP (12.0) and the embedded raster
is itself at least MIN_NATIVE_MP_FOR_CAP (4.0). Capping at native resolution
is not downsampling — the render then carries exactly the pixels the source
image holds.

⚠️ Do NOT key this on dpi alone. An earlier version capped at the page's
native dpi unconditionally. That is catastrophically wrong for pages whose
image is small but stretched over the page: those measure a native dpi as low
as 5 (a 40 px graphic on an A4 page), and capping to 5 dpi would render a
41x58 px page, destroy OCR and leak PII silently. Their 150 dpi render is only
~2 MP — nowhere near the ceiling. 42 of 593 measured corpus pages embed an
image under 2 MP. Upscaling is suppressed only when the image is already big
enough that upscaling buys nothing.

Measured distribution over 593 pages of a real 投资者适当性 corpus: 517 pages
render at 2-5 MP, 6 at 5-12 MP, and only 28 exceed 12 MP — so the cap trips on
~5% of pages and leaves the rest byte-identical. Coverage:
tests/test_render_dpi.py.

Output naming is Chinese, and configurable (changed 2026-09-02)

Redacted copies land in a 已脱敏/ folder inside the input folder, and each
file carries a _已脱敏 suffix. The delivery folder and that suffix are the
only things telling a reviewer "this copy is safe to send out", so they are in
the language the people handling the files actually read.

Both are overridable — the previous English names still work:

PRIVACY_REDACTOR_OUTPUT_DIR=已脱敏        # delivery folder name
PRIVACY_REDACTOR_OUTPUT_SUFFIX=_已脱敏    # "" to drop the suffix entirely

This is functional, not cosmetic. Folder mode writes inside the source
tree, so a re-run walks over its own previous output and would otherwise
re-redact redacted files or nest 已脱敏/已脱敏/. The engine recognises its own
output by this suffix; every site that names an output and every site that
skips one reads from scripts/naming.py so they cannot drift apart. Setting
the suffix to empty switches that skip to a path-based exclusion of the output
folder instead.

The suffix is also what stops safe_output_path() from ever writing a redacted
file over its own source.

BIRTHDAY is gated on the page being a credential scan (narrowed 2026-09-02)

detect_birthday only runs on pages that already carry a detected ID-card
number, on the reasoning that a printed date there is the holder's birthday.
That gate was the wrong one, and this is the distinction that matters:

> Quoting an ID number is not the same as being an ID document.

Every 投资者适当性 form quotes the investor's ID number (投资者证件编号：…) while
also printing half a dozen business dates — assessment timestamps, statement
dates, investment-history periods. Presuming all of those were birthdays
blacked out ~100 business dates across one corpus, including every 测评时间
— and the risk-assessment timestamp is exactly what an appropriateness review
needs in order to judge whether an assessment is still within its validity
period. Losing it is pure loss bought with no privacy.

Measured on that corpus: zero ID-card scans. Of ~100 dates caught on
ID-bearing pages, not one was a birthday. Sampling the survivors after an
earlier partial fix confirmed the rest were 证件有效期 (2044.5.3), 成立日, and
statement dates sitting behind money figures.

The rule now requires one of:

• an explicit 出生 / 生日 label near the date — taken on any page; or
• the page carrying a credential marker — 公民身份号码 / 签发机关 / 居民身份证 /
  有效期限. These appear on an ID card and on nothing else; a form uses
  证件号码 / 证件编号 instead, which is what makes them a clean discriminator.

Dates that are positively business — behind 测评时间 / 签署 / 签订 / 打印 / 制表 /
成立日 / 有效期, in the future, opening a validity range, or right behind a
money figure — are skipped outright.

Accepted residual risk: if a real ID-card page loses its 公民身份号码 /
签发机关 markers to a bad OCR pass, an unlabelled birthday there would be
missed. The mitigation is that the ID number itself is still MASKed, and the
birth digits embedded at positions 7-14 of an 18-digit ID go with it.
Coverage: tests/test_birthday_narrowing.py.

Unlabelled long digit runs are now redacted (changed 2026-08-31)

The verifier has always failed any surviving run of >= 15 digits
(LONG_DIGIT_RE), on the reasoning that a number that long is essentially
always an identifier. But nothing redacted them, so the engine routinely
produced output that its own gate rejected.

From 2026-08-31 the detector claims them too: any run of >= 15 digits not
already taken by another pattern becomes a BANK_CARD (so the policy's REDACT
applies). LONG_DIGIT_RE is deliberately defined identically in
regex_detector.py and security/verifier.py so detector and gate cannot
disagree.

⚠️ This reverses the 2026-08-29 decision that a bare long number should be
left to the verifier "to keep false positives low". What forced the change:
the scanned-page text layer let the verifier read rasterised output for the
first time and it failed 10 files straight away. A survey of every >= 15 digit
run in the corpus (32 total) found 29 claimed by no detector at all:

• 18 fund account numbers in table cells — the column header says 基金账号 but
  the individual cells carry no adjacent label, so ACCOUNT_LABEL_RE (which
  needs 账号/账户号 directly before the digits) cannot anchor on them;
• 2 bank cards made Luhn-invalid by an OCR digit slip;
• 3 证照编号;
• 1 run holding two phone numbers OCR-merged together
  (021-5550123413900001234) — LANDLINE_RE needs a non-digit after its 7-8
  digits and MOBILE_RE needs one before its 11, so the concatenation defeats
  both;
• plus product codes and flow numbers (see the cost note below).

Cost, measured: ~5 of the 32 are business identifiers rather than personal
PII (product codes like ZXD202601070000000001, flow numbers like
2025021200000001). They now get redacted too. The engine cannot tell a
基金账号 from a 产品编码 by shape, and the standing preference is to err
toward redaction. If a delivery needs product codes legible, say so and this
can be narrowed.

Two guards: a run adjacent to * is skipped (readable MASK leaves a long
prefix, e.g. 913100006315******, and that prefix can itself satisfy Luhn);
and anything already claimed by ID_CARD / BANK_CARD / ACCOUNT_LABEL_RE wins.
Coverage: tests/test_corporate_account.py.

OCR line merging inserts a space between apart blocks (fixed 2026-08-31)

ImageAdapter._merge_to_lines used to concatenate the blocks on a line with no
separator at all. That silently welds separate form fields together, and the
weld defeats every detector's boundary guard. Measured on a real 基金合同:

• "效期" + "2038.01.02" + "021-55501234" became
  "效期2038.01.02021-55501234" — LANDLINE_RE's leading (?<!\d) then
  rejects the phone, because the character before its leading 0 is the 2
  of "02";
• another line produced "021-5550123413900001234", a landline welded to a
  mobile, which fails LANDLINE_RE's (?!\d) and MOBILE_RE's (?<!\d)
  at the same time.

Blocks are now joined with a space when the horizontal gap exceeds
LINE_GAP_CHAR_RATIO (1.5) times the previous block's character width.

The threshold must leave split numbers alone — this merge exists precisely
so a RapidOCR-split ID ("11010119900307" + "0003") is reassembled, and those
blocks are pixel-adjacent. Measured gap distribution over 114 samples from 8
pages: 0 samples in 0-0.5 char widths (where split numbers live), 3 in
0.5-1.0, 27 in 1.0-2.0 (adjacent label/value pairs), 83 at >=2.0. The two
failing lines measured 14.17 and 14.95, so 1.5 clears both populations.
Coverage: tests/test_scanned_text_layer.py.

Rasterised pages keep an invisible text layer (added 2026-08-31)

Redacting a scanned page paints over a raster and re-packages it, so the output
page has no text layer. The verifier reads the text layer only
(extract_text → page.get_text("text")), so on such a page it saw an empty
string, found nothing, and reported PASS without having verified anything.
Measured on a real 139-file corpus: 19 of 126 output PDFs were in that state,
including a 141-page contract.

The engine now embeds the recognised text back as an invisible layer
(render_mode=3), so the output stays searchable and verifiable.

⚠️ The replacements are applied BEFORE embedding — this is not optional.
Writing the raw OCR text would hide PII behind a black box on screen while
leaving it fully readable to anyone who copies text or runs text extraction —
materially worse than not redacting at all. Blocks covered by a redaction are
dropped and the entity contributes only its replacement, mirroring exactly
what was painted.

Cost, measured: ~0.1 s per dense page against 1.6-3.8 s for that page's OCR
(0.7-2.1% of a full batch), and +14 KB per file — the font is referenced,
not embedded. (TextWriter is ~18x faster but embeds a full CJK font at
+1.6 MB per file; subsetting it would need fontTools. Not worth it.)

Related verifier fix: remaining_original_entities used a plain substring
test. OCR routinely clips the last character off an entity (a BIRTHDAY read as
2025-06-2 where the page's printed date is 2025-06-24), and the clipped
value is then a prefix of a value that legitimately stays — which raised a
false "N original value(s) still appear" on every such page. It now requires a
clean boundary (the next character must not be alphanumeric). Coverage:
tests/test_scanned_text_layer.py.

Emptied <AppVersion> made Word refuse redacted DOCX (fixed 2026-09-12)

A customer reported that a redacted .docx would not open in Word at all:

文件可能已经损坏。The file may be corrupted.

Nothing visible was wrong with it. The ZIP container was intact
(testzip() clean, 86/86 parts present), every XML part was well-formed, all
relationships and Content_Types resolved, python-docx opened it happily, and
the engine's own verifier said PASS. The whole file was dangerous precisely
because every automated check liked it.

How it was found — bisection against the real application. Word 16 was
driven through COM (Word.Application → Documents.Open), and the failure was
reproduced on the customer's own file. Then a package was rebuilt from the
original with ONE part swapped in from the broken output at a time:

Narrowing inside that part to a single element each time:

Root cause. The adapter's metadata sanitiser treated the authoring-tool
fingerprint like the other metadata fields — it emptied the element's text
but kept the tag. <AppVersion></AppVersion> is invalid: extended-properties
wants a version string shaped like 16.0000. Word escalates that single empty
element into "the whole package is corrupt" and refuses to open it. Note the
asymmetry that made this so quiet: blanking is fatal, removing is
harmless.

Template / Application are blanked harmlessly, but they are dropped as
well — they identify the authoring tool, and an absent element hides it more
thoroughly than an empty one.

Why no test caught it: the fixture in tests/test_docx.py hand-writes an
app.xml containing Application and Company but no AppVersion — the
fatal branch was never exercised. The new fixture in
tests/test_docx_app_props.py is verbatim the shape Word 2016 writes,
AppVersion included, and the regression is mutation-checked (re-introducing
the blanking makes it fail).

Repairing files that already shipped. Files redacted before this fix carry
the same defect and will not open, so they do not need re-redacting — only the
three elements removed:

python scripts/fix_docx_app_props.py --input <file-or-folder>          # scan only
python scripts/fix_docx_app_props.py --input <file-or-folder> --apply  # repair + backup

It is scan-only unless --apply is given, keeps a timestamped backup of every
file it edits, preserves entry order, and re-checks each file afterwards.
Coverage: tests/test_docx_app_props.py.

Lesson for any future OOXML surgery: "the parser accepts it" is not the
bar. Word applies schema validation on top of XML well-formedness, and the
only trustworthy oracle is Word itself — a redacted DOCX is not proven good
until Word has opened it.

License (L1 soft lock)

The engine is licensed with a first-run activation + 60-day trial window:

• The skill ships with no per-client configuration. The first time any
  engine script (redact.py / detect.py / verify.py) runs on a machine, it
  activates itself by writing a license file to
  ~/.workbuddy/licenses/privacy-redactor.json (env override:
  PRIVACY_REDACTOR_LICENSE_DIR).
• Every later run verifies the activation time; after 60 days the engine
  refuses to work and exits with code 5 printing a
  {"status": "LICENSE_EXPIRED" ...} JSON.
• Tampering with activated_at in the license file is detected (HMAC
  signature) and also rejected with code 5.

When the engine exits with code 5, tell the user plainly that the trial /
license has expired and no redaction was performed. Do not attempt to
bypass, edit, or delete the license file — that is prohibited. The customer
must renew through the vendor.

Step 3 — Semantic review handoff (smart mode ONLY)

If the engine exits with code 4, it printed a SemanticReviewRequest JSON
to stdout. This contains only the fuzzy candidates (e.g. "张主任", "王经理")
with their text, context, candidate_type, and confidence — never the raw
document. For each item, decide:

{ "index": 0, "sensitive": true, "type": "PERSON" }

Write the decisions as a SemanticReviewResponse JSON
({"schema_version":"1.0","decisions":[...]}) and re-invoke:

python SKILL_DIR/scripts/redact.py \
  --input file --policy /path/policy.json \
  --review-response /tmp/review.json --mode smart

Smart mode on a FOLDER (batch AI intervention, one round trip)

--mode smart with a folder input now works through two stages — the engine
defers every file with fuzzy candidates and asks the AI to judge the whole set
ONCE (not per file). Customer-facing names: 敏感内容定位（定位阶段）→ AI 脱敏（脱敏阶段）;
the engine-internal labels pass 1 / pass 2 below are kept only for AI ↔ engine
protocol compatibility:

• 定位阶段 (engine pass 1, no --review-response): files without
  candidates are redacted directly; files with candidates are DEFERRED.
  No masking happens for deferred files in this stage — it is pure
  locating. When the walk finishes the engine prints ONE folder-level
  request to stdout and exits with 4:

{"schema_version":"1.0","mode":"smart","scope":"folder","batch":true,
 "files":[{"file":"<rel/path>","items":[{index,text,context,
           candidate_type,confidence},...]}, ...]}

• The AI judges every item and writes a folder-level response (keys are the
  relative paths from the request):

{"schema_version":"1.0",
 "files":[{"file":"<rel/path>",
           "decisions":[{index,sensitive,type},...]}, ...]}

• AI 脱敏阶段 (engine pass 2): re-invoke with --review-response
  pointing at that response and the same --report path. Only the
  deferred files are re-processed (with the AI's decisions applied — this is
  where the actual masking happens); all other files are carried over from the
  定位阶段 report, so the final combined report covers the whole folder and
  gains a review block (reviewed_files / total_candidates).

# 定位阶段 (exit 4 expected)
python SKILL_DIR/scripts/redact.py --input folder --output folder/已脱敏_smart \
  --mode smart --scope folder --ner heuristic \
  --focus-list "脱密重点关注清单.docx" --report /tmp/smart.json

# AI 脱敏阶段 (after the AI writes /tmp/smart_review.json)
python SKILL_DIR/scripts/redact.py --input folder --output folder/已脱敏_smart \
  --mode smart --scope folder --ner heuristic \
  --focus-list "脱密重点关注清单.docx" --report /tmp/smart.json \
  --review-response /tmp/smart_review.json

Notes: --focus-list composes with smart folder mode (SKIPPED files never
reach the review step). In the 定位阶段 the engine persists progress to
--report before exiting 4, so an AI 脱敏阶段 run always has the full
picture.

⚠️ Candidate explosion (measured, now auto-mitigated). The bundled
heuristic NER is intentionally conservative and flags thousands of word
fragments as needs_review (a 121-file focus folder produced 130,885
candidates — ~118k PERSON, of which random sampling showed ~95% are fragments
like 金管理/万元/任何法, NOT real names). Smart folder mode now applies an
automatic review-workload filter (scripts/review_filter.py, enabled in
smart mode unless --no-review-filter):
- Dedup — identical (type, text) candidates in a file collapse to one
review item; apply_review propagates the verdict to all copies.
- Filename whitelists — investor names parsed from filenames are NEVER
filtered (they are the real PII); fund/product names parsed from filenames
(牛福号, 私募证券投资基金) are treated as non-PII fragments.
- Word/structure checks — PERSON fragments (common words like 万元/范围, or
surname+common-word like 金管理) and ORGANIZATION sentence fragments /
public-org suffixes (银行/证券/公司/总局…) are dropped. ADDRESS is never
filtered.
- Context (方案3) — HeuristicNERDetector now fills review_context (the
±10-char window around each entity), which both reaches the AI in the review
request AND drives context rules: name-field hints (投资者/姓名/签字/身份证…,
后接 先生/女士/签字) keep the candidate; 基金X patterns (preceded by 基金/
本基金/该基金) or function-word tails drop it.

Measured on the same 121-file folder, tier progression:
130,885 → 23,865 (dedup, -82%) → 23,795 (+whitelist) → 11,761 (+words, -91%)
→ 8,058 (+context, -93.8%), projected AI review time ~5.2 h → ~19 min,
with identical pass-2 results (0 residual, same known hard-case FAIL). See
redact_work/review_filter_tiers.md for per-tier filtered examples.

Step 4 — Verify, then report (MANDATORY)

After redaction, the engine runs the verifier automatically and the printed
AuditReport contains a verification.status:

• PASS — all checks passed.
• PASS_WITH_WARNINGS — passed but with warnings (e.g. OCR unavailable, or a
  low-confidence entity could not be re-verified). Surface the warnings to the
  user.
• FAIL — some high-confidence PII or custom term remains, or metadata was not
  removed.

The verifier also carries a long-digit safety net (added 2026-08-29): any
numeric run of ≥15 digits still present in the output fails the gate
regardless of what the detectors reported. It exists because 对公 (corporate)
bank accounts used to slip through — they fail the Luhn checksum and can exceed
the 19-digit cap of BANK_RE, so they were never detected and the verifier
silently reported PASS. Readable-MASK artefacts (320211************) and
allowed values are excluded, so correct output is not flagged.

Corporate bank accounts are detected by LABEL, not checksum (added
2026-08-29): ACCOUNT_LABEL_RE matches 账号 / 帐号 / 账户号 / 账户号码
(full- or half-width colon, optional spaces) followed by 12–30 digits and
requires no Luhn check. A label-anchored account also outranks any USCC
that merely overlaps it, so the first 18 digits of a 26-digit account can never
be claimed as a USCC (which would have masked only the last 6). Personal cards
still go through the original Luhn-validated BANK_RE path.

You may tell the user the task succeeded ONLY when verification.status is
PASS or PASS_WITH_WARNINGS. When it is FAIL: NEVER auto re-run or
auto-tighten the policy in a loop. List the residual + full file path and
report it — the customer decides whether to hand-fix. Measured repeatedly:
most FAILs are stamp/edge hard cases (印章遮挡) that NO policy change or
re-run can fix; an AI "repairing" them blocks the whole delivery. An automatic
fix is allowed AT MOST ONCE per file, and ONLY when the residual type is
clearly policy-fixable (e.g. a policy term/regex the engine simply missed).
If it still FAILs after that one attempt, STOP that file immediately and move
on — never enter a fix → re-verify → fix loop.

⚠️ Customer-friendly residual wording (MANDATORY). NEVER use the word
"FAIL / 失败 / 校验不通过" in customer-facing messages — it is alarming and
inaccurate for what is usually a tiny artifact. The engine already appends a
soft Chinese note to the audit warnings (e.g. "校验发现 N 处残留（多为印章遮挡
或黑块边缘外露），建议人工补涂确认；其余内容已通过校验。"). Mirror that tone:
- Few residuals (typically ≤ 5, stamp/edge artifacts like a URL tail under a
seal): say "有 N 处微小残留（多为印章遮挡或黑块边缘外露），建议人工补涂确认，
其余已全部通过校验" — no FAIL word, no alarm.
- Large-scale residuals (many hits / core fields largely exposed): say
"存在明显残留，需人工确认处理" — still no FAIL word, but be clear the
customer must act on it. The AI does NOT auto re-run these in a loop; it
reports them once and stops.
Internally you still treat the machine status as the single source of truth
(never claim success on a FAIL), but the wording you hand the customer is
calm and actionable.

⚠️ Prohibited: Never claim success by visually inspecting the output file
or by trusting that a redaction "looks" complete. The verifier's machine
output is the single source of truth. Do not assert "已完全脱敏" unless the
verifier returned PASS.

⚠️ Files the engine could not read MUST be relayed (MANDATORY). Any batch
report carries a skipped_unsupported block whenever the input folder held
files whose format the engine cannot read:

"skipped_unsupported": {"count": 5, "needs_conversion": 2,
  "files": [{"file": "旧表.xls", "reason": "needs conversion",
             "hint": "Legacy binary Excel format. Re-save it as .xlsx ..."}]}

Name every entry to the customer, and pass on its hint. Never report the
batch as finished without mentioning them: an unsupported format is the ONE
failure a customer cannot spot for themselves — the folder looks processed,
overall says PASS, and the skipped file appears nowhere else. Two reasons:

• reason: "needs conversion" (.xls .ppt .pptx .rtf .odt) — the
  file is readable in principle, just not in this format. Say how to re-save it
  (the hint does) and offer to run it once converted.
• reason: "unsupported format" (.ods .heic .dwg, …) — a hard no.
  Say so plainly, and name what IS supported.

If overall is FAIL and the report carries an incomplete block reading
"nothing was processed", the entire folder was unreadable: do not describe
the run as done, and never leave an empty 已脱敏/ folder without explaining why
it is empty. Those files are also listed in files[] with
status: "SKIPPED", so the report remains a complete inventory of the input.

Scratch files are a separate block. ~$<name>, .~lock.<name># and *.tmp
are Office/LibreOffice owner locks, not documents — Word and Excel leave one
behind while a file is open. They arrive under skipped_temporary, are skipped
rather than processed (a run must NOT go FAIL because someone left Word open),
and are still named so nothing vanishes silently. Do not present them as
documents that were processed or failed; mention them only if it explains why a
file the customer expected to see handled is absent.

A human-readable report lands in the delivery folder. Every folder run
writes 已脱敏/处理报告.md at the top level of the output — counts, the
per-file table, every file that was NOT processed, elapsed time, the masking
rules in plain language, and the limits of this run. It is written
unconditionally (unlike the JSON report, which needs --report) because it is
what the person who opens the folder reads. Point the user at it when you
report a batch: it saves you re-typing the inventory, and it travels with the
deliverable. Do not contradict it — if you need to say something the report
does not, it is probably a gap worth telling them about.

已脱敏/未脱敏/ holds the inputs the engine could NOT read, copied across
unchanged so the delivery accounts for every file the customer sent. These are
NOT sanitized. A .pdf that would not open may contain exactly the PII the run
was meant to remove, and a file sitting inside a folder called 已脱敏 invites
the assumption that it is clean. The folder name and the report both say
otherwise; say it too when you hand a batch over — and if the customer only
wanted the sanitized set, suggest deleting that subfolder before the delivery
leaves the building.

Long-running tasks: proactive progress reporting (MANDATORY)

When processing a folder with many files, or compressing large videos, the job
can take 20+ minutes. The user must NEVER be left waiting silently. Follow
this rule:

1. Always tell the user the job is running and give an expected duration
   ("正在处理 298 个文件，预计 30-60 分钟") before starting. Say that the run
   adds the brand marker and how to turn it off — the overlay is on by default
   and announced (see "Output branding overlay" above), so it belongs in this
   message too. Do not let it be discovered later. Progress reporting is about
   the job.
2. Stage naming in reports (MANDATORY). In every progress update and
   summary, call the stages 敏感内容定位（定位阶段） and AI 脱敏（脱敏阶段） —
   never "pass 1 / pass 2". When the 定位阶段 finishes, explicitly tell the
   customer that locating is done and actual redaction has NOT happened yet
   ("定位完成：已标记 N 个文件含需确认的敏感内容，尚未脱敏；接下来进行 AI 判定并脱敏").
   The 脱敏阶段 is where masking happens.
3. Live progress file (the customer's eyes). The engine writes
   <output>/progress.json after EVERY finished file: {stage, mode, workers, total, done, pct, current, elapsed_sec, eta_sec, updated_at}.
   stage tells the customer which phase they are in: locating (敏感内容
   定位中 — 尚未脱敏), locating_done (定位完成，等待 AI 判定 — 仍未脱敏),
   redacting (AI 脱敏中), done (全部完成). Its mtime and content change
   continuously while the job runs. MANDATORY (fallback guarantee): at the
   start of every real (正式) run, tell the customer progress.json's location
   and that it updates in real time — "任务已开始，可随时打开输出目录下的
   progress.json 查看实时进度；即使对话播报有延迟，这里也是实时更新的。"
   Even if the AI's chat updates stall, the customer can always self-check.
   Point the user to it at start ("您可以随时打开 output/progress.json 查看
   实时进度") — this is the guarantee that progress is VISIBLY moving even
   between AI reports. progress.json is a
   runtime-only file: the engine removes it when the final run finishes. The
   engine does NOT write .audit.json / .allowed.json sidecars by default —
   the audit lives at the --report path (always pass one for folder runs),
   and sidecars are only produced with the explicit --write-allowed flag. So
   the delivery folder contains ONLY the sanitized documents, 1:1 with the
   source folder.
4. Report every ~10 minutes — from the progress file, not from memory. Do
   NOT wait for a completion notification alone. For any task expected to
   take more than ~10 minutes, poll progress.json / the engine log every
   ~10 minutes and post a short update to the chat: "已处理 150/292（51%），
   预计还需约 30 分钟". Every progress report must be backed by the current
   progress.json numbers so the customer can cross-check.
5. Never go silent. If the AI cannot self-wake, the fallback is the
   progress file itself: tell the user up front where it lives so they can
   watch it even if chat updates stall. A silent wait of many hours with zero
   updates — even with the job running fine — is a failure of this skill.
6. On interruption or timeout (process killed, session switched, system
   slept): detect what finished vs. what did not (compare source files against
   existing _已脱敏 / _compressed outputs / progress.json), resume the
   unfinished part, and tell the user exactly what was recovered.
7. Persist results to a log file (the engine stderr log next to outputs)
   so progress survives session restarts and can be inspected later.
8. When the job finishes, deliver the full report (success/skipped/failed +
   sizes) as described in Step 4 — proactively, WITHOUT waiting for the user
   to ask or for the next user message. The final report must cite the final
   --report summary (overall + per-file verification). Note: progress.json
   and sidecars are auto-cleaned at completion, so capture the final numbers
   from the --report JSON, not from progress.json.

Polling mechanism (concrete steps — self-paced, NEVER wait for the user)

You CAN and MUST poll inside the same turn — do NOT wait for the user to
send another message, and do NOT treat "no completion notification yet" as a
reason to stay silent. Use sleep-based slices in your own loop:

• Launch it as a MANAGED background task — never with a shell nohup … &.
  A shell-detached child is reaped the moment that tool call returns: the run
  dies after a handful of files and then looks exactly like "stuck on a big
  scanned PDF" (log stops growing, progress.json freezes, output folder still
  holds only the previous run's files). Use the agent runtime's own background
  mechanism so the process stays owned by a tracked task.
• Right after launch: confirm the process is alive and already writing
  outputs — check progress.json exists under the output folder, tail the
  engine log.
• If a run freezes early with tiny progress, suspect the process before the
  file. Check it still exists and check its memory footprint: a live
  OCR/scanning run holds hundreds of MB, so a python process sitting at ~10-25
  MB means nothing is actually running and the job must be relaunched (reuse the
  same --output, already-written files are simply overwritten).
• Then loop (no user input needed): sleep in SHORT slices (60-120s — a
  short, fast-returning tool call keeps the loop alive on models that tend to
  stall on long commands) and after EVERY slice re-read
  <output>/progress.json:
• report ONLY on a meaningful change: pct rose ≥10 points, or stage /
  done changed (this caps chat updates at ~10 per run). When reporting,
  POST a short update ("已处理 150/292（51%），预计还需约 30 分钟") citing
  the current progress.json numbers so the customer can cross-check;
• if nothing changed, stay quiet and sleep the next slice;
• if the process exited, break out of the loop.
• Detect completion by polling, then report immediately: completion is
  signalled when outputs stop growing AND the process has exited AND the
  --report audit JSON exists — NOT by a notification and NOT by the user
  asking. The instant this is observed, deliver the full report
  (success/skipped/failed + residual PII) proactively.
• NEVER say "I'll wait for it to finish and report back" and then go
  silent. A silent multi-minute wait while the job runs — even if it finishes
  successfully — is a failure of this skill.

Focus-list filtering (process only the categories that matter)

When the user wants to redact only a subset of files in a large folder
(e.g. a "脱密重点关注文件" checklist), pass --focus-list pointing at a
checklist document. The engine parses the checklist into categories, then
skips (does NOT process) any source file whose name does not fuzzy-match a
category — this often cuts the workload by ~50%.

Parallel folder processing (--workers N): folder mode runs
serially by default (1 process). Whether to OFFER parallelism at all is
decided by the size gate above (only folders whose pdf+doc files are
mostly ≥2MB get the --workers 4 question). On multi-core machines,
--workers 4-8 runs files in parallel (each worker loads its own OCR
model, ~1GB RAM each; a 32-core / 32GB machine comfortably fits 4-8).
Measured:
- 10-file scanned-contract subset (one 86-page scan dominates): 1w 522s /
2w 445s (1.17x) / 4w 462s (1.13x, WORSE — CPU oversubscription) /
4w with per-worker ONNX thread cap 397s (1.31x).
- Full 问题附件 folder, smart mode (299 files, 75 big scans): 2:36:33 →
1:39:57 with 4 workers (1.57x), identical results (250 PASS + 42 WARN +
6 ~$ ERRORs, 0 FAIL / 0 residual; 292 sanitized docs).

Hard lessons: (1) --workers>1 MUST cap ONNX threads (auto:
cores // workers, total ≤ cores) or N OCR processes slow each other down;
(2) when ONE huge scan dominates, total time is floored by that file
(Amdahl) — parallel gains are largest on folders with MANY big scans;
(3) do NOT size-sort tasks (measured worse — big files then contend for CPU
together). Scheduler is a sliding window (never submit-then-wait — that
serialises). Timeout kills/restarts the pool and re-queues unfinished files.
--workers composes with smart folder two-pass mode (AI review happens
between the two engine passes, unaffected by worker count).

python SKILL_DIR/scripts/redact.py \
  --input folder --output folder/已脱敏_focus \
  --mode smart --scope folder --ner heuristic \
  --focus-list "260822脱密重点关注文件.docx" \
  --report /tmp/focus_audit.json

• Checklist formats: .docx (python-docx, with zipfile fallback), .txt
  (one category per line), .json ({"categories":[...]} or a bare list).
  Rows that are titles (end with ：/:) or too short (≤1 char) are ignored;
  numbered rows like 1..6 are dropped.
• Matching is on the file NAME only (fast; it does NOT open files to OCR
  their content). This is intentional — the goal is efficiency.
• Keyword → category rules (priority = match order; lower wins first so
  the most specific category claims the file):
  1. 风险匹配告知书 / 风险匹配 → 投资者风险匹配告知书
  2. 基本信息表 → 投资者基本信息表
  3. 基金合同 / 合同签署 / 补充协议 → 基金合同（PDF版，PDF + Word 都处理）
  4. 告知确认书 / 告知书 / 风险告知书 → 投资者告知确认书（含"风险告知书"归此）
  5. 申购 / 认购 → 申购单，认/申购申请书
  6. 身份证 / 身份证件 / 证件照 → 身份证
• Everything else is SKIPPED (招募说明书、合格投资者承诺函、税收居民声明、
  风险测评/风险揭示书、募集回访确认函、证明类、投资者查询账号开通等).
• Audit reporting: each processed file records focus_category; the
  combined report gains a focus_filter block with
  {"categories":[...], "matched":N, "skipped":M} so the savings are
  self-evident. Skipped files appear with status:"SKIPPED".
• Fuzzy points must be confirmed with the user before the first run (e.g.
  does "基金合同（PDF版）" also cover 合同签署页 / 补充协议? does "风险告知书"
  belong to 告知确认书 or 风险匹配告知书? does "身份证" include 证件照?). The
  4 decisions above are the agreed defaults — adjust keywords_for_category()
  in scripts/focus_filter.py if the user changes their mind.

Handling failures

Golden rule: NEVER loop on FAILs. A FAIL file gets AT MOST ONE automatic
fix attempt; if it still fails, STOP that file, list it in the report and
move on. Do NOT "repair" FAILs by re-running over and over, tightening the
policy repeatedly, or re-OCR'ing the same file — measured experience: most
FAILs are stamp/edge hard cases (印章遮挡) that no re-run or policy change can
fix, and an AI looping on them blocks the whole delivery.

• Exit 1 (processing error: corrupt PDF, OCR crash, etc.): read the
  printed error, fix the specific input/environment problem, retry ONCE. If
  it fails again, stop and report — never retry the same file more than twice
  in total.
• Exit 2 (verification FAILED): the audit's verification block lists
  what remained. Default action: do NOT fix automatically. Report the
  residual + full file path (customer wording per Step 4) and let the customer
  decide on manual touch-up. Only if the residual is clearly policy-fixable
  (e.g. a policy term/regex the engine simply missed) do ONE targeted fix;
  still FAIL → stop that file, report it, move on. Never loop.
• Exit 3 (unsupported format): tell the user the file type is not
  supported in this version. No retry.

Capabilities & limits (be honest with the user)

• Supported now: PDF (native / scanned / mixed, per-page classification),
  DOCX / Word .docx + .docm (OOXML surgery on body, headers/footers,
  footnotes, endnotes, comments, text boxes, tables, hyperlink targets, and
  document metadata; track-changes revisions are normalized first),
  PNG/JPG/JPEG/WebP, TXT, Markdown, CSV / TSV
  (scripts/adapters/csv.py, added 2026-08-30 — see the paragraph below),
  XLSX / XLSM (Excel 2007+; scripts/adapters/xlsx.py, added 2026-09-13 —
  see the XLSX paragraph below).
  Regex + checksum detection for ID card,
  phone, email, bank card (Luhn), IP, URL, USCC; pluggable local NER (bundled
  heuristic) for PERSON/ORGANIZATION/ADDRESS; consistent pseudonyms; real PDF
  redaction (apply_redactions); image pixel redaction; metadata cleanup;
  electronic seals / digital signatures preserved by default
  (preserve_signatures: true — business seals stay and are reported as an
  informational PASS_WITH_WARNINGS note; the verifier only hard-fails on a
  leftover signature when the policy explicitly set preserve_signatures: false, in which case PKCS#7 / /FT/Sig fields are stripped. Seal
  annotations (/Subtype /Screen) are excluded from the metadata check by the
  same rule that reports them, so the two checks can never disagree);
  video compression to a target size (two-pass H.264,
  scripts/video_compress.py, target configurable in config/compress.yaml);
  automated verification.
• Requires an optional dependency to improve: OCR (rapidocr / easyocr /
  paddleocr / pytesseract) for images and scanned PDFs, and a model-backed NER
  to replace the heuristic detector. Without OCR, image / scanned-page PII is
  not detected (the engine warns, it never silently skips). With an OCR
  backend present, embedded raster images inside DOCX (word/media/*.png|jpg|…)
  are OCR'd and redacted just like standalone images.
• Legacy .doc (pre-2007 binary Word): SUPPORTED via text extraction
  (scripts/adapters/doc_legacy.py, requires olefile). Text-level redaction
  is applied and the result is rebuilt as a .docx (paragraph structure kept;
  fonts/tables/images are not round-tripped). Encrypted .doc files cannot
  be read without a password — the engine reports this and asks the user to
  re-save as .docx in Word. DOCX produced by LibreOffice/WPS/Google Docs is
  fine.
• Honest limits on DOCX: embedded raster images (word/media/*.png|jpg| jpeg|bmp|gif|tiff|webp) ARE OCR'd and redacted when an OCR backend is
  available. Vector formats (EMF / WMF / SVG) inside word/media/ cannot be
  OCR'd and are left untouched with a warning. VBA macros, customXml, and OLE
  objects are NOT parsed (the engine prints a warning, it never silently skips).
• Roadmap (interfaces reserved, not yet implemented): PPTX/HTML/
  JSON adapters; advanced replacement strategies.

CSV / TSV specifics (added 2026-08-30)

• The whole sheet is flattened into one string with a NUL separator between
  cells, detected once, then spliced back using the original row widths —
  so row/column structure is preserved exactly, including ragged rows and
  quoted cells that contain commas or newlines.
• NUL is the separator precisely because no PII pattern admits it: a number
  split across two neighbouring cells can never be stitched into a fake ID or
  phone number. Matches that still straddle a boundary (possible via the
  [\s　]* in ACCOUNT_LABEL_RE) are dropped, because replacing one
  would merge two cells and shear the row.
• Encoding: input is decoded via utf-8-sig → gb18030 → utf-8 → big5 → latin-1 (mainland-Windows Excel still writes GBK). Output is written as
  UTF-8 with BOM so Excel renders Chinese correctly on double-click.
• Delimiter: chosen by column-count consistency over the first 50 rows,
  not by csv.Sniffer — Chinese product names contain commas and confuse it.
• No brand marker line is prepended to CSV/TSV (unlike TXT/MD): it would
  displace the header row and break every downstream parse.
• Measured on a real 329-row inventory sheet: 248 ID numbers + 78 credit codes
  detected, 0 residual, structure identical, 6 AI review candidates.

XLSX / XLSM specifics (added 2026-09-13)

• Detection is the existing rules, unchanged. No XLSX-specific detector
  exists or is needed: the sheet is flattened into one NUL-separated string
  (the same trick, and the same separator constant, as CSV) and handed to the
  normal pipeline. Header rows, a summary block sitting under the detail block,
  ragged rows and empty columns never reach detection, so the adapter needs no
  notion of a "table block" and no column-header rules.
• Shared strings are never edited in place. Cell text lives in
  xl/sharedStrings.xml and cells reference it BY INDEX, so several cells can
  point at one <si>. A changed cell gets a NEW <si> appended (deduplicated
  by value) and its index repointed, because the unit of redaction is one
  occurrence — the review layer can rule on two occurrences of the same string
  differently. If the workbook has no sharedStrings part, the cell is written as
  t="inlineStr" rather than synthesising that part, its content-type override
  and its relationship.
• Formula cells are skipped and reported. A cell can carry <f> plus a
  cached <v>; replacing only the cache would let Excel recompute the original
  on open and put the PII straight back. Existing policy has no rule for
  formulas, so these cells produce a warning — never a silent pass.
• A masked numeric cell becomes text, so the cell type changes and
  arithmetic referring to it stops working. Reported when it happens.
• NOT rewritten in this version, and warned about when it matters: cell
  comments, sheet names, defined names, and sheet header/footer text. If
  anything the policy would act on sits there, the run says so instead of
  passing over it.
• Metadata is sanitised by the same shared routine the DOCX adapter uses
  (security.metadata.sanitize_ooxml_props): author, last-modified-by,
  Application, AppVersion, and TitlesOfParts (which leaks sheet names) all
  go. Only parts that actually changed are re-serialised; every other zip entry
  is copied byte-for-byte.
• .xls (binary BIFF) is still not supported — save as .xlsx first.

USCC detector fixes (2026-08-30)

Two bugs found by a full-corpus run; both are fixed in
scripts/detectors/regex_detector.py:

1. Charset dropped X. USCC_RE used [0-9A-HJ-NP-RTU-WY], which parses
   as 0-9|A-H|J-N|P-R|T|U-W|Y — no X. Any credit code carrying X in
   positions 9-17 matched neither USCC_RE nor the truncated pass (which
   skips full 18-char values), so it leaked. One such code was reported PASS
   by the verifier: it has letters, so the pure-digit long-run net never saw
   it. Now [0-9A-HJ-NP-RTUWXY] — the true GB 32100 31-character set.
2. Checksum missed the complement. GB 32100 is
   C18 = 31 - (Σ(Ci×Wi) mod 31); the code returned _USCC_CHARS[total % 31],
   so validate_uscc() was False for every input and gave zero
   discrimination. Now _USCC_CHARS[(31 - total % 31) % 31].

Both are covered by tests/test_csv_adapter.py (including the regression that
18-digit ID numbers still win the overlap and are never reported as USCC).

The bundled heuristic NER is intentionally conservative and flags candidates as
needs_review. Treat its output as a hint, not ground truth.

内部调试：其他引擎模式（不得用于客户文件）

> 本小节仅供内部与后台测试使用。
> 客户侧一律使用 smart（见「Mode routing」开头的 CLIENT-FACING RULE）。
> 未取得明确的内部测试授权，不得对客户文件运行下列任何模式。

三种模式按 AI 复审工作量从左到右递增、误涂从左到右递减（均已验证 0 残留）：

（2026-08-22 曾试验"模型化 NER（jieba 词性）"作为方案4并入 balanced，实测 jieba
把大量常用词当人名、且无法区分"金管理"与"莫志宏"，属净噪声——已删除。若未来引入
真模型（HanLP/LAC 级别），接口在 detectors/ner_detector.py 的
get_ner_detector() 处预留。）

--mode balanced 与 --mode smart 都会应用 review-filter；strict_local 完全
不应用。

在 strict_local 下引擎从不发起复审请求：模糊或低置信度实体直接按策略处理，并
作为 warning 标记；原始文档正文也不会发送给 AI。这与 smart 的两阶段流程形成
对照——需要 AI 逐条判定时，请用 smart。