AI Short Video Creation — Photo + Video Motion Transfer

Transfer a reference video's motion onto the subject(s) in a photo to produce a
new short video. Core engine: built-in VideoGen image-to-video, driven by a
motion prompt distilled from the reference video's key frames.

Inputs

If either required input is missing, ask the user to provide it before proceeding.

Workflow

1. Confirm inputs. Verify the photo and reference video paths exist. If the
   user only described the idea without attaching files, ask them to attach both.
2. Extract reference frames. Run scripts/extract_frames.py on the reference
   video to get evenly-spaced key frames (default 8). This lets you see the motion.
3. Analyze motion. Read the extracted frame images. Follow
   references/motion-analysis.md to extract action type, body regions,
   trajectory, rhythm, key poses, and energy level. Record the result as a
   structured motion description.
4. Describe the subject. Read the photo(s). Identify person count, appearance
   (clothing, hair, build, age range), pose, and background — these must be
   preserved in the output.
5. Build the generation prompt. Combine subject description + motion
   description using the matching template in references/prompt-templates.md
   (single-person, upper-body, two-person, or multi-person mapping).
6. Generate the video. Load the VideoGen tool via ToolSearch, inform the
   user of the credit cost (~50-100 credits for a 5s video), then call it with
   the photo as the input image and the constructed prompt. See "Calling VideoGen".
7. Present the result. Use present_files with the generated video file.
   Summarize the motion that was transferred and any deviations.
8. Offer refinement. If motion fidelity is insufficient, offer to regenerate
   with an adjusted prompt or recommend an external service (see step in
   "Escalation" below).

Extracting Reference Frames

Run the bundled script (use the managed Python runtime):

"<managed-python>" "<skill-dir>/scripts/extract_frames.py" "<video-path>" --count 8 --width 720

• It prints one absolute frame path per line. Read each frame image to analyze motion.
• For fast/complex motion, increase --count to 12-16.
• If ffmpeg is missing and OpenCV is unavailable, install one, or fall back to
  analyzing the video directly if the runtime supports reading video files.

Calling VideoGen

VideoGen is a deferred tool. Load then invoke it:

1. ToolSearch with tool_names: ["VideoGen"] to load the schema.
2. Before calling, tell the user: video generation costs ~50-100 credits per
   5-second clip, and ask for confirmation to proceed.
3. DeferExecuteTool with toolName: "VideoGen" and params:
   - prompt: the constructed motion+subject prompt from step 5.
   - image: path to the subject photo (image-to-video mode).
4. The result is saved as a local MP4. Present it with present_files.

Multi-Person Handling

When the photo or reference video contains more than one person:

1. Count and label each person in the photo, left-to-right (Person A, B, ...).
2. Map motions: if the reference video also has multiple people, map
   reference performers to photo subjects left-to-right by default; confirm the
   mapping with the user if ambiguous.
3. Describe spatial relationships in the prompt (facing each other, side by
   side, one behind) and synchronization (together / mirroring / alternating).
4. Single generation is preferred (one prompt covering all people) for
   coherence. Only split into separate generations if the model struggles with
   the group and the user agrees.
5. See references/prompt-templates.md Template 3 & 4 for multi-person wording.

Escalation to External Services

The built-in VideoGen produces an approximation of the reference motion, not a
frame-accurate copy. When the user needs precise pose/choreography fidelity:

• Consult references/external-services.md for the service shortlist.
• Recommend Viggle AI first (purpose-built image+video→video motion transfer).
• Present the option and let the user choose; never auto-submit the photo to an
  external service without explicit consent.

Limitations

• Motion reproduction is approximate — rhythm and key poses are captured, but
  not exact frame-by-frame pose matching.
• Very fast, occluded, or extremely complex multi-limb motion may diverge.
• Generation length is typically 5s; longer sequences need stitching or looping.
• Background changes may occur despite "preserve background" instructions.
• Do not generate content that impersonates real people without consent, or
  reproduces copyrighted choreography/performances.

Resources

• scripts/extract_frames.py — extract key frames from the reference video.
• references/motion-analysis.md — how to analyze frames into a motion description.
• references/prompt-templates.md — generation prompt templates by scenario.
• references/external-services.md — external motion-transfer services for precision.