Joint Modeling of Text, Audio, and 3D Motion Using RapVerse

This article introduces RapVerse, a large-scale dataset and framework that enables the joint generation of 3D whole-body motion and synchronized rap vocals directly from textual lyrics. By scaling autoregressive transformers across language, audio, and motion modalities, the authors demonstrate compelling results in multimodal music generation. While currently limited to the rap genre, the framework holds promise for broader applications in virtual performances and live AI-driven concerts. Future directions include support for multi-performer scenarios and expansion into other musical styles.