A Single Prompt Will Have This AI Rapping and Dancing

This paper introduces RapVerse, a novel dataset and unified AI framework that simultaneously generates realistic singing vocals and full-body 3D motion directly from text lyrics. Leveraging a multimodal transformer trained on synchronized lyrics, vocals, and 3D mesh data, the system advances beyond traditional siloed approaches by merging language, audio, and motion into a seamless autoregressive generation pipeline. Extensive experiments show that this joint generation model performs competitively with specialized single-modality systems, setting a new benchmark for text-to-performance AI.