Talking Head Editor

> 由 未迟AI笔记（公众号：未迟AI笔记）维护并分享。关注获取更多 AI 视频提效实战技巧。

Overview

End-to-end pipeline for editing talking-head / 口播 videos:

1. Transcribe the video with Whisper (faster-whisper, base, CPU, Chinese).
2. Auto-suggest edits: remove meaningless fillers (呃/额/嗯嗯), compress long pauses
   (>0.8s → 0.3s), drop retakes / false starts.
3. Interactive editor (web UI on http://localhost:8765): preview the video with a
   karaoke subtitle overlay, manually toggle/adjust cut segments, compress gaps, edit
   subtitle text, pick subtitle color, then render the final clip.
4. Burn subtitles (karaoke per-word coloring) and export the final video.

The skill's defining behavior: every invocation launches the editor first so the user
can verify the auto-cuts and tweak subtitles before any rendering happens. Nothing is
burned into the video until the user clicks "渲染视频".

What's new in v1.2.0

• Portable venv resolution — no hardcoded machine paths; the interpreter is auto-detected.
• Typo-correction helper (correct_transcript.py) — Whisper mis-hears Chinese homophones
  a lot; this rebuilds word timestamps after you fix the text (no API key needed).
• Horizontal + vertical sources — OUTPUT_WIDTH/HEIGHT now documented as "match the real
  frame size" so burned subtitles never distort.
• Fullscreen inside restricted iframes — verified to work in the WorkBuddy preview pane.
• One subtitle-size knob, two sync points — documented so preview == burned size always.

When to Use

• User wants to trim / de-hesitate / add subtitles to a talking-head or 口播 video.
• Trigger phrases: 剪辑口播视频, 视频去口癖, talking head edit, 口播稿剪辑, 给口播加字幕.
• Output target is short-form vertical (1080×1920, 9:16) by default, but any aspect works
  if you set OUTPUT_WIDTH/OUTPUT_HEIGHT to the source frame (see Pitfalls #6).

Environment & Dependencies

• Python venv (required for faster_whisper + flask):
• config.VENV_PYTHON auto-detects the interpreter in this order:
  1. A venv bundled next to the code: <skill>/scripts/talking-head-editor/.venv/
     (create once: python -m venv .venv then .venv/Scripts/pip install -r requirements.txt).
  2. WorkBuddy managed python venv: ~/.workbuddy/binaries/python/envs/talking-head/Scripts/python.exe
     (or bin/python on Linux/macOS).
  3. python3 / python on PATH — but it MUST already have faster_whisper and flask.
• No hardcoded absolute path is used, so the skill runs on any machine. Where this doc
  says PY, substitute the resolved venv python above.
• Code location (self-contained inside this skill): scripts/talking-head-editor/.
  All modules resolve paths relative to their own file, so WORK_DIR becomes
  scripts/talking-head-editor/workspace/ automatically. No external path assumptions.
• ffmpeg: auto-detected by config._find_ffmpeg() — prefers imageio_ffmpeg's binary,
  then workspace/ffmpeg_path.txt, then PATH, then WinGet. Works out of the box.

Workflow (run every time)

All commands below use PY = the resolved venv python above, and TK =
scripts/talking-head-editor (relative to this skill directory). WS = TK/workspace.

Step 1 — Prepare the preview source

Copy the user's video into the workspace as input_subbed.mp4.

• If the source is HEVC/H.265 (common from phones/剪映), first transcode to H.264 so
  the browser preview can play it AND so the frame is clean (no pre-burned subtitles):

bash "$PY" - <<'PYEOF' import sys; sys.path.insert(0, r"<TK>") from config import FFMPEG_PATH from pathlib import Path SRC = r"<path-to-user-video>" OUT = r"<TK>/workspace/input_subbed.mp4" cmd = [FFMPEG_PATH, "-y", "-i", SRC, "-c:v", "libx264", "-crf", "20", "-preset", "medium", "-c:a", "copy", OUT] import subprocess; subprocess.run(cmd, check=True) PYEOF

• If the source is already H.264/MP4, just copy it to WS/input_subbed.mp4.

> The preview source MUST be subtitle-free. Never feed a video that already has burned-in
> subtitles — the editor would then show two subtitle layers (see Pitfalls).

Step 2 — Transcribe & auto-suggest cuts

"$PY" "$TK/main.py" "<user-video>" --language zh

This writes <stem>_transcript.json and <stem>_edit_decisions.json into WS/.
Then rename/copy them to the editor's expected fixed names:

cp "$WS/<stem>_transcript.json"      "$WS/input_subbed_transcript.json"
cp "$WS/<stem>_edit_decisions.json"  "$WS/input_subbed_edit_decisions.json"

(<stem> = video file name without extension.)

> Windows long-task note: Whisper on CPU can take minutes. If you launch main.py
> directly and the WorkBuddy session later goes idle, the process tree may be recycled and
> the transcription killed. Prefer running it in the background (or detached via Task
> Scheduler) so it survives idle, then poll the produced JSON. The editor server is lighter
> and usually fine, but apply the same caution for long sessions.

Step 2.5 — Fix transcription typos (optional but recommended for Chinese)

Whisper is full of Chinese homophone errors (e.g. 油箱→邮箱, 霸格→bug, 采坑→踩坑,
维持→未迟). The helper correct_transcript.py applies your fixes and rebuilds the
per-word timestamps the karaoke effect needs.

1. Export one line per segment:

bash "$PY" "$TK/correct_transcript.py" export "$WS/input_subbed_transcript.json" segs.txt

2. Edit segs.txt — fix each line's text. Keep the line count identical (one line =
   one segment, in order); a blank line keeps that segment's original text.

3. Apply and write back (original is backed up to *.bak):

bash "$PY" "$TK/correct_transcript.py" apply "$WS/input_subbed_transcript.json" segs.txt

This is a pure heuristic — no API key. For LLM-assisted correction, paste segs.txt to any
model, get the corrected lines back, and apply. (Rebuilding words loses original per-word
probabilities, but the karaoke highlight still works.)

Step 3 — Launch the interactive editor

"$PY" "$TK/editor_server.py"

Open http://localhost:8765 in the browser. The editor shows:

• Left: video preview + timeline (deleted segments red, compressed gaps blue) + stats
  (original vs. final duration, compression %).
• Right transcript panel: click a sentence to seek; double-click to delete/restore;
  double-click a gap bar to compress/keep; "字幕编辑" to edit subtitle text per line;
  "字幕颜色" dropdown to pick cyan/yellow/red/blue/green/white/orange.
• Bottom subtitle preview: karaoke style — unspoken words gray, spoken/current words in
  the chosen color. No bounce, no background block (matches the burned output exactly).
• ⛶ 全屏: requests fullscreen on the video container; if the Fullscreen API is blocked
  (e.g. inside a restricted preview iframe), it falls back to a CSS "fake fullscreen" that
  still fills the window.
• 🎬 渲染视频: renders the final clip.

> If port 8765 is taken, kill the old process (Get-NetTCPConnection -LocalPort 8765) and
> restart. Restart the server whenever render.py / editor_server.py / video_processor.py
> / editor.html / config.py change, so the new code is loaded.

Step 4 — Render

The user clicks "渲染视频" in the editor. render.py writes the result to
WS/input_subbed_render.mp4 — never back to input_subbed.mp4 (see Pitfalls). The
editor then offers preview/download of the final clip.

Background detail: render() cuts segments (frame-accurate re-encode, NOT -c copy which
snaps to keyframes and silently keeps gaps), burns the karaoke ASS subtitle, and exports.

Key Features (v1.2.0)

• Karaoke subtitles, consistent across preview and output.
• Editor: per-word coloring driven by Whisper word-timestamps (already in the transcript).
• Final video: ASS subtitle with \kf per-word fill tags (gray → chosen color).
• Font size scales with video width via a single ratio — but it lives in two places
  that must stay in sync (see Pitfalls #5): config.SUBTITLE_FONT_RATIO (burned) and
  editor.html JS const SUBTITLE_FONT_RATIO (preview). Current value 0.05 →
  ~54px on a 1080-wide frame, so preview and burned are visually identical at any window.
• Colors: cyan (default) / yellow / red / blue / green / white / orange, selectable in the
  editor and applied to both preview and burned output.
• Fullscreen that survives restricted iframes. The editor requests fullscreen on the
  .video-wrap container (not document), with a CSS fallback when the API is blocked.
  Don't regress this — it's what makes "全屏" work inside the WorkBuddy preview pane.
• Horizontal & vertical sources. Set config.OUTPUT_WIDTH/OUTPUT_HEIGHT to the real
  frame size (see Pitfalls #6); render.py does NOT resize, so the ASS PlayRes must match
  or subtitles distort. Default is vertical 9:16 (1080×1920).
• Typo-correction helper (correct_transcript.py) — rebuild word timestamps after
  fixing Whisper's homophone errors, no API key required.
• Auto-edit suggestions preserved: filler removal, pause compression, retake/false-start
  dropping. The editor always loads these so the user can accept/tweak them.
• Editor-first: no rendering happens without the user's explicit action.

Known Pitfalls (read before "fixing" anything)

1. Double subtitles (双字幕). Root cause: an older render.py wrote the burned result
   back to input_subbed.mp4, overwriting the clean preview source. The editor then stacked
   its own karaoke layer on top of the burned-in (often stale/uncut) subtitle → two subtitles,
   one large (burned, wrong content) and one small (karaoke, correct). Fix already applied:
   render.py now writes to input_subbed_render.mp4 and never touches input_subbed.mp4.
   If you ever see double subtitles again, regenerate a clean input_subbed.mp4 (Step 1) and
   make sure nothing writes back to it.
2. HEVC sources. The browser preview cannot play HEVC; also, if the source already has
   burned subtitles, the editor will double them. Always produce a clean H.264 input_subbed.mp4.
3. Keyframe snapping in cuts. Using -ss before -i with -c copy snaps each cut to the
   nearest keyframe and silently keeps the gap you meant to delete (final duration ≈ original).
   cut_segments is already fixed to post--i -ss/-to with re-encode for frame accuracy.
   Do NOT "optimize" it back to stream copy.
4. Whisper is CPU/slow. base model on CPU takes a few minutes for a 1–2 min clip. Reusing
   a cached transcript avoids re-running it. Word timestamps are already enabled in transcribe()
   (needed for karaoke) — do not disable. Long runs launched directly by the agent on Windows
   may be killed when the session idles — run them in the background / detached (see Step 2 note).
5. Subtitle size is ONE knob in TWO places. Changing config.SUBTITLE_FONT_RATIO alone does
   NOT change the preview: editor.html sets the preview size via a JS inline style
   (subPreviewEl.style.fontSize = videoWidth * SUBTITLE_FONT_RATIO), which overrides any CSS.
   Always change both config.SUBTITLE_FONT_RATIO and the editor.html JS constant together.
   Current synced value: 0.05. (If you ever see "preview huge but burned tiny" or vice-versa,
   this mismatch is the cause.)
6. Output size must match the frame. render.py does NOT resize, so config.OUTPUT_WIDTH/ OUTPUT_HEIGHT must equal the source frame, or the burned ASS subtitle (PlayRes) will stretch.
   Vertical 9:16 → 1080×1920; horizontal 16:9 → 1920×1080; other aspect → set to the source.
   Re-check this whenever you switch between a vertical and a horizontal clip.
7. Venue of edits. This skill directory is the authoritative copy. Keep changes here. Do NOT
   hand-edit a dated session-folder copy of the code — changes there won't ship and will be lost.

File Map (under scripts/talking-head-editor/)

• main.py — pipeline entry (transcribe → suggest → optional burn).
• transcribe.py — Whisper transcription (word timestamps on).
• video_processor.py — cut/burn; generate_ass_karaoke() builds the karaoke ASS.
• render.py — render pipeline; render() honors cuts + burns input_subbed_render.mp4.
• editor_server.py — Flask server serving editor.html + JSON APIs (port 8765).
• editor.html — the interactive editor UI (karaoke preview, color picker, cut/compress UI,
  container-level fullscreen with CSS fallback).
• edit_engine.py — cut/compress decision logic.
• correct_transcript.py — export segment texts / apply fixes / rebuild word timestamps.
• config.py — paths, venv auto-detect, Whisper params, subtitle font ratio + color palette.
• requirements.txt — faster-whisper (+ flask for the editor, imageio-ffmpeg for ffmpeg).
• workspace/ — runtime scratch (input_subbed.mp4, _transcript.json, _edit_decisions.json,
  input_subbed_render.mp4). Starts empty; populated per session. Never commit user videos here.