Virtual Singer MV Production

Problem

Creating a virtual-singer MV usually requires aligning lyrics to beats, generating scene footage, keeping the character consistent, animating the singer, and compositing subtitles with the original track. This skill targets a concrete production case where the inputs include music audio, lyric text, a video script or storyboard, and a virtual singer three-view reference image. It organizes the workflow into staged, user-confirmable steps rather than relying on one opaque model call.

How It Works

The pipeline first analyzes audio with openai-whisper-api and scripts/analyze_audio.py, extracting BPM, beat timestamps, and duration to create lyrics_timeline.json and scene_timeline.json. It then generates scene clips with VideoGen from text or concept images, optionally supplementing material with media-downloader. Character images are produced from the three-view reference using ImageGen, then refined and upscaled with nano-banana-pro. Animation can use VideoGen image-to-video, video-fx stylized templates, or image-to-3D model generation. Finally, remotion-video-toolkit composites scenes, character animation, original audio, and lyric subtitles into a 1080P video, with FFmpeg as a fallback if rendering fails.

Boundaries

The skill is intended for scene-based MVs, not general music generation or lyrics-only creation. If the audio is mainly instrumental, Whisper may not produce usable lyrics, so supplied lyric text becomes important. Failures in VideoGen, 3D generation, or Remotion rendering can fall back to still images, template animation, or FFmpeg assembly. Character consistency depends on repeated key visual descriptors, and the default output is 1080P at 30 FPS.