Short Video From Audio
Overview
Take one background image, one audio narration, a title string, and produce a
ready-to-publish 9:16 mp4 short video with title at the top, time-stamped
Chinese subtitles in the mid-lower area (auto-generated from the audio via
whisper.cpp), and the original audio track. Sub titles are baked-in drawtext
frames so the output needs no further processing before upload.
When to use
- "我要做短视频" + 图片 + 音频 + 标题 → 视频
- 公众号/小红书 配套视频内容生产
- 任何"语音 → 带字幕的竖屏短视频"场景
Workflow
Step 1 — Confirm and stage input files
The user provides:
- An audio file (mp3 / m4a / wav). Confirm with ffprobe that the file plays.
- A background image (png / jpg, ideally portrait ~9:16). If landscape, confirm
whether to scale-and-crop or add blurred side margins.
- A title string.
Build a project directory like /Users/jx/Downloads/<topic>-p<n>/.
Step 2 — Prepare audio for ASR
ffmpeg -y -i INPUT.mp3 -ar 16000 -ac 1 -c:a pcm_s16le audio.wav
Step 3 — Run whisper.cpp ASR (Chinese)
~/.local/bin/whisper-cli \
-m ~/models/ggml-small.bin \
-l zh \
--prompt "..." \
-f audio.wav \
-osrt -of subs \
--entropy-thold 1.5
If 复利 / 复盘 / similar tokens appear mis-transcribed as 负利 / 反盘, run
sed -i '' 's/负利/复利/g' subs.srt and back up the original as subs.raw.srt.
Step 4 — Generate per-segment drawtext chain
Run the bundled script to convert SRT → one drawtext filter per subtitle:
python3 scripts/srt2drawtext.py subs.srt [shift_seconds] > drawtext_chain.txt
This produces ~150-300 lines for a 9-minute audio. The script supports:
- Time shift: pass a seconds value to drop early segments (e.g. podcast intro)
and rebase all timestamps to t=0. Used with the trimmed audio so subtitles stay
aligned. Example: python3 scripts/srt2drawtext.py subs.srt 9.680 > chain.txt.
- Auto-wrap: any subtitle > 13 chars is wrapped into multiple lines at
punctuation or word boundaries (hard-cut fallback). Critical — drawtext does
NOT honor \n in the text option on macOS, so wrapping must be done by emitting
multiple drawtext filters at different y positions, not by inserting newlines.
- Adaptive font size: 68 / 64 / 60 / 56 / 52 / 48 px for 1/2/3/4/5/6 line blocks
(large, readable style — subtitle block occupies ~1/5 of the screen).
- Mid-lower anchored stacking: block bottom fixed at y=1500 (NOT the very
bottom — keeps subtitles clear of platform progress bars / action bars at the
bottom of the screen), grows upward.
Do not put a trailing , on each line — the build script will join them
with paste -sd ','.
Step 5 — Compose final mp4 with ffmpeg
See full command in references/notes.md (titled "Full ffmpeg invocation pattern").
Key parameters:
- canvas 1080×1920
- framerate 30
- libx264 -preset medium -crf 22
- aac -b:a 192k
- -movflags +faststart
Title style (top, shifted down ~80px to leave headroom): main title in 2 lines
fontsize=88 (y=140/266), subtitle in 3 lines fontsize=60 (y=550/636/722),
both in the same yellow #FFE100 with black border (borderw 3-4) + semi-transparent
black box (boxborderw 12-16) for readability on busy backgrounds. Subtitle style:
fontsize=68/64/60/56/52/48 based on line count, white text + black border +
semi-transparent black box, block bottom-anchored at y=1500 (mid-lower, NOT
bottom), enable='between(t,start,end)'.
Step 6 — Spot-check
ffmpeg -i output.mp4 -ss <n> -frames:v 1 -q:v 2 /tmp/check.jpg at t=5s,
t=audio_mid, t=audio_end. Verify Chinese renders correctly, title doesn't
clip, subtitles appear on time.
Resources
scripts/
srt2drawtext.py— converts an SRT file to drawtext filters per segment,
withenable='between(t,...)'for time-windowed rendering. Critical for
Chinese rendering on macOS where ffmpeg'ssubtitlesfilter fails. Invoke as
python3 scripts/srt2drawtext.py input.srt [shift] > drawtext_chain.txt.
Supports time shifting and auto-wrap for long subtitles.
references/
notes.md— pitfalls, verified toolchain fingerprints, exact drawtext style
blocks, and the canonical full ffmpeg command. Read this before debugging
any "box squares for Chinese characters" or "title gets truncated" issue.
