This AI Turns Lyrics Into Fully Synced Song and Dance Performances

This article presents a novel benchmark and model for generating both singing vocals and full-body motion directly from textual prompts like rap lyrics. By aligning these two modalities during training, the model surpasses state-of-the-art baselines in vocal quality, motion realism, and synchronization (measured via metrics like BC, FID, and LVD). It outperforms cascaded approaches like DiffSinger + Talkshow while reducing computational overhead. Ablation studies reveal the importance of modality-specific VQ-VAEs and the limitations of generic large language models for multimodal generation. This work marks a major step forward in text-driven AI performance synthesis.