Text-to-Rap AI Turns Lyrics Into Vocals, Gestures, and Facial Expressions

This article introduces a novel generative AI framework that transforms written rap lyrics into synchronized vocal performances and full-body animations—including gestures and facial expressions. Built on the RapVerse dataset, the system uses a unified token representation for text, audio, and motion. Through a modular architecture of motion and vocal tokenizers, and a large autoregressive model, it creates expressive, multimodal outputs from lyrical input—bridging the gap between natural language and animated performance.