Local Video Subtitle Extraction & Proofreading

Turn a local video into a clean, readable subtitle/script. Three extraction paths,
one rule-based proofread layer, and an optional hard-subtitle comparison that fixes
ASR mis-hearings using the video's burned-in text.

When to use

• "提取这个视频的字幕" / "把视频转成文字" / "字幕校对"
• A local video file (mp4/mkv/webm/...) with either embedded subtitles, burned-in
  (hard) subtitles, or just speech.
• No API keys, no cloud — everything runs locally.

Environment (read first)

• Python 3.10+ with these packages: faster-whisper, pysubs2,
  opencc-python-reimplemented, pillow. Install once (see references/setup.md for
  the China mirror + HuggingFace env vars):
  bash pip install faster-whisper pysubs2 opencc-python-reimplemented pillow
  Run every script with that Python, e.g. python scripts/transcribe.py ....
  (On the author's machine a dedicated venv at
  C:/Users/Administrator/.workbuddy/binaries/python/envs/subtitle already has them.)
• ffmpeg must be on PATH (provides ffmpeg + ffprobe). Scripts auto-detect them;
  if missing they print an install hint and exit. Always pass Windows paths
  (E:/视频/x.mp4); native ffmpeg ignores /e/ msys paths.
• Set before any model download / run:
  bash export HF_ENDPOINT="https://hf-mirror.com" export HF_HUB_DISABLE_XET="1"
• Known pitfalls (GPU specifics, OCR download blocked, xet 401, etc.) → references/lessons.md.
  Format/ffmpeg recipes → references/formats.md.

Workflow

Step 0 — Decide the path (probe first, don't assume)

1. Probe for embedded subtitles (Path A):
   bash PYTHON="<venv>/Scripts/python.exe" "$PYTHON" scripts/extract_embedded.py "E:/视频/x.mp4" out.srt
   - Exit 0 → embedded subtitle extracted. Done (skip to Step 4 for proofread/export).
   - Exit 2 → no embedded subtitle track. Continue to Step 1.
2. Check for burned-in (hard) subtitles: extract one frame and Read it
   (scripts/extract_frames.py with --count 1, or ffmpeg -ss 30 -frames:v 1).
   If hard subtitles are visible, the user chooses ASR (recommended) or skip.

Step 1 — ASR transcription (Path B, the reliable default)

"$PYTHON" scripts/transcribe.py "E:/视频/x.mp4" raw.srt --model small --lang zh

• Works on any video/audio. Auto-extracts 16k mono wav (cached as raw.srt.wav).
• --lang auto detects language; zh forces Chinese. small is a good balance;
  medium/large-v3 for higher accuracy (slower).
• Device: --device auto (default) uses CUDA if an NVIDIA GPU is present, else
  CPU. On an NVIDIA GPU pass --device cuda for a large speedup (CTranslate2
  supports CUDA natively — no extra setup beyond an NVIDIA driver).
  On Intel Arc (A770 etc.) use --device cpu — the Arc GPU cannot produce valid
  timestamps via OpenVINO (see references/lessons.md §4), but CTranslate2's CPU
  mode runs fine and is exactly what this skill was tested on (an A770 machine).

Step 2 — Rule-based proofread (always)

"$PYTHON" scripts/proofread.py raw.srt cleaned.srt

• Traditional→Simplified (OpenCC), strips empty cues, merges duplicate consecutive
  lines, sorts by time. Fully offline, no LLM.

Step 3 — (Optional) Hard-subtitle comparison & correction

Fixes ASR mis-hearings (反制主义→反智主义, etc.) by reading the video's burned text.
1. Extract frames:
bash "$PYTHON" scripts/extract_frames.py "E:/视频/x.mp4" frames/ --count 16 --collage
2. Read the frames with vision (Read tool on each frames/f*.jpg, or the
collage.jpg). Note the correct wording where ASR clearly mis-heard.
3. Build a correction map fixes.json — a JSON list of [wrong, right] pairs, e.g.
[["反制主义","反智主义"],["反制","反智"],["优技","优绩"],["公立","功利"]].
Optionally a pending.json list of {term,pos,guess,note} for items needing
human review.
4. Apply + report:
bash "$PYTHON" scripts/correct_srt.py cleaned.srt fixes.json \ --out-srt fixed.srt --out-txt fixed.txt --report report.md \ --title "视频标题" --pending pending.json \ --copy-srt "C:/Users/Administrator/Desktop/字幕_校正版.srt"

Step 4 — Export readable document

"$PYTHON" scripts/export_doc.py cleaned.srt \
    --txt "C:/Users/Administrator/Desktop/字幕文稿.txt" \
    --title "视频标题" \
    --copy-srt "C:/Users/Administrator/Desktop/字幕.srt"

Produces a paragraph-merged plain-text transcript and copies the SRT alongside.

Scripts reference

Known limitations

• NVIDIA GPU (CUDA): fully supported — pass --device cuda (or auto). This is
  the fastest path. CTranslate2 bundles CUDA support; only an NVIDIA driver is needed.
• Intel Arc (A770 etc.) — GPU acceleration unavailable, but CPU mode works fully.
  The Arc GPU cannot produce valid timestamps via OpenVINO (it drops the
  cross-attentions timestamps depend on — see references/lessons.md §4), so do not
  use it as an accelerator. CTranslate2's CPU mode runs perfectly on an Arc machine —
  in fact this skill was developed and tested on an Intel Arc A770 (the bundled
  examples originated there). Run with --device cpu.
• AMD GPU (ROCm): not supported by the current backend. faster-whisper's engine
  (CTranslate2) ships CUDA + CPU wheels only — there is no ROCm build. On AMD the
  script falls back to CPU (clean timestamps, slower). A DirectML / whisper-ort backend
  could cover AMD/Intel/NVIDIA on Windows but is not implemented yet and untested
  here (see references/lessons.md §4). Contributions welcome.
• Hard-subtitle OCR via easyocr is not viable in China (model weights undownloadable;
  see references/lessons.md §5). Use multimodal frame reading instead.
• Rule-based proofread does not fix semantic errors — that needs the Step 3
  hard-subtitle comparison (or an LLM, out of scope by default).