Towards Unified Song Generation and Singing Voice Conversion with Accompaniment Co-Generation
0. Contents
1. Abstract
While song generation and singing voice conversion (SVC) have evolved significantly, they have long been developed isolated: the former lacks zero-shot speaker cloning, while the latter overlooks vocal-accompaniment synergy. To bridge this gap, we propose UniSinger, the first end-to-end framework unifying speaker cloning song generation and accompaniment co-generation SVC. Building on the multimodal diffusion transformer, we construct a unified speaker embedding space transferring speaker representation from SVC to song generation, endowing fine-grained cross-task timbre control. To mitigate multi-task optimization conflicts, we design a curriculum learning strategy using task-specific modality masking to guide the model to gradually master the generative mechanisms among semantic content, vocal timbre, and accompaniment. Experiments demonstrate state-of-the-art performance on both tasks and realizes complementary benefits, offering new possibilities for intelligent music production.
Fig.1 The architecture of UniSinger.
2. Samples on Song Generation
| Instruction | Transcript | Reference Audio | Synthesis Audio |
|---|---|---|---|
|
genre: The genre of music is hip hop. instrumentation: The instruments used in the music piece include a drum machine, a synthesizer, and a bass guitar. singer: The music is performed by one singer. The singer is male, in his 20s, with a confident and energetic tone. rhythm and melody: The tempo is 100 BPM, and the key signature is G major. The melody is characterized by a strong, rhythmic beat and a catchy, repetitive hook. mood and atmosphere: The music feels like a lively, upbeat party scene. The combination of the confident male vocals, the driving drum machine beat, and the catchy synth melody creates a sense of excitement and energy, as if the singer is inviting the listener to join in the fun." |
I thought I better hurry up before I busted a groove. I leaned down to kiss her but then out of the blue. A door slammed and a voice said baby where are you. | ||
|
genre: The genre of music is soul and blues. instrumentation: The instruments used in the music piece include a guitar, a bass, a piano, and a drum set. singer: The music is performed by one singer. The singer is a male in his 40s. The tone of the singer is soulful and emotional. rhythm and melody: The tempo is moderate, around 75 BPM. The key signature is F minor. mood and atmosphere: The music evokes a sense of melancholy and introspection. The soulful guitar riffs and the bluesy piano chords create a somber atmosphere, while the singer's emotional tone adds a layer of vulnerability and longing. The overall mood is reminiscent of a rainy day, with the music acting as a soothing balm for the listener's soul. |
What you let me do? what I want to you? | ||
|
genre: The genre of music is traditional Chinese. instrumentation: The instruments used in the music piece include a guzheng, a dizi, and a yangqin. singer: The music is performed by one female singer. The singer is in her 40s with a soft and melodic tone. rhythm and melody: The tempo is moderate, and the key signature is B major. The melody is soothing and harmonious. mood and atmosphere: The music evokes a sense of tranquility and elegance, reminiscent of a serene garden on a quiet afternoon. The gentle strumming of the guzheng, the delicate notes of the dizi, and the soft plucking of the yangqin create a peaceful and reflective atmosphere, perfect for relaxation or meditation. |
不要陶醉梦境美. | ||
|
genre: The genre of music is classical. instrumentation: The instruments used in the music piece include a piano and a female voice. singer: The music is performed by one singer. The singer is a female in her late teens to early twenties. Her tone is soft and ethereal. rhythm and melody: The tempo is moderate, with a steady and flowing rhythm. The key signature is D major. mood and atmosphere: The music evokes a sense of tranquility and introspection. The soft piano and the ethereal female voice create a dreamlike atmosphere, as if the listener is floating on a cloud. The combination of the gentle melody and the subtle dynamics of the performance invite the listener to lose themselves in a peaceful and reflective state. |
Restless heart beats so imperfectly. | ||
|
genre: The genre of music is classical. instrumentation: The instruments used in the music piece include a piano and a choir. singer: The music is performed by one singer. The singer is a male in his 30s. The tone of the singer is passionate and emotive. rhythm and melody: The tempo is moderate, with a steady and flowing rhythm. The key signature is A major. mood and atmosphere: The music evokes a sense of reverence and solemnity. The combination of the piano's gentle melody and the choir's harmonious voices creates a profound and uplifting atmosphere, reminiscent of a spiritual or ceremonial moment. |
哦亲爱的党,为了民族的兴旺 | ||
|
genre: The genre of music is pop. instrumentation: The instruments used in the music piece include a piano, synthesizer, and a drum machine. singer: The music is performed by one female singer. The singer is in her 20s, and her tone is soft and melodic. rhythm and melody: The tempo is moderate, around 170.0 bpm. The key signature is A minor. mood and atmosphere: The music evokes a sense of nostalgia and melancholy. The soft piano and electric guitar create a dreamy, ethereal atmosphere, while the drum machine adds a subtle, rhythmic foundation. The singer's tone, combined with the overall arrangement, conveys a feeling of longing and introspection. |
醉人的香味,谁能了解? | ||
|
genre: The genre of music is pop. instrumentation: The instruments used in the music piece include a piano, electric guitar, and a drum set. singer: The music is performed by one singer. The singer is male, in his 30s, with a smooth and melodic tone. rhythm and melody: The tempo is moderate, around 140 BPM, and the key signature is F minor. mood and atmosphere: The music evokes a sense of longing and introspection, with the singer's smooth voice and the gentle strumming of the guitar creating a melancholic yet hopeful atmosphere. |
你的心,近在天边,却离我,那么远. |
3. Samples on Sing Voice Conversion
| Instruction | Source Audio | Reference Audio | Synthesis Audio |
|---|---|---|---|
|
genre: The genre of music is hip hop. instrumentation: The instruments used in the music piece include a drum machine, a synthesizer, and a bass guitar. rhythm and melody: The tempo is 100 BPM, and the key signature is G major. The melody is characterized by a strong, rhythmic beat and a catchy, repetitive hook. mood and atmosphere: The music feels like a lively, upbeat party scene. The combination of the confident male vocals, the driving drum machine beat, and the catchy synth melody creates a sense of excitement and energy, as if the singer is inviting the listener to join in the fun." |
|||
|
genre: The genre of music is pop. instrumentation: The instruments used in the music piece include a piano, synthesizer, and electric guitar. rhythm and melody: The tempo is moderate, around 140 BPM, and the key signature is F minor. mood and atmosphere: The music evokes a sense of longing and melancholy, with the singer's smooth voice and the soft, dreamy instrumentation creating a reflective and emotional atmosphere. |
|||
|
genre: The genre of music is pop. instrumentation: The instruments used in the music piece include a guitar, a bass, and a drum. rhythm and melody: The tempo is 134.033 bpm. The key signature is D major. mood and atmosphere: The music feels like a heartfelt confession, with the singer's emotional tone and the soft, melodic guitar and bass creating a sense of vulnerability and longing. |
|||
|
genre: The genre of music is pop. instrumentation: The instruments used in the music piece include a guitar, a keyboard, and a drum set. rhythm and melody: The tempo is moderate, around 100 BPM, and the key signature is Bb major. mood and atmosphere: The music creates a warm and nostalgic atmosphere, evoking feelings of love and longing. The soft guitar strums and gentle keyboard melodies blend harmoniously, accompanied by the rhythmic drum beats, to create a soothing and intimate listening experience. |
|||
|
genre: The genre of music is classical. instrumentation: The instruments used in the music piece include a piano, strings, and a choir. rhythm and melody: The tempo is moderate, with a steady and flowing rhythm. The key signature is F minor. mood and atmosphere: The music evokes a sense of tranquility and introspection. The soft piano melodies, combined with the gentle strings and choir, create a serene and reflective atmosphere, reminiscent of a peaceful countryside village. |
|||
|
genre: The genre of music is hip hop. instrumentation: The instruments used in the music piece include a prominent drum beat, a bassline, and a synthesizer. rhythm and melody: The tempo is moderate, around 90 BPM, with a steady and driving rhythm. The key signature is A minor. mood and atmosphere: The music feels energetic and upbeat, with a sense of confidence and determination. The combination of the rhythmic drum beat, the smooth bassline, and the confident vocal delivery creates an atmosphere of empowerment and ambition. |
|||
|
genre: The genre of music is hip-hop. instrumentation: The instruments used in the music piece include a prominent bassline, rhythmic drums, and a synthesizer. rhythm and melody: The tempo is moderate, around 140 BPM, with a steady beat that drives the rhythm. The key signature is C minor, giving the music a somber yet energetic feel. mood and atmosphere: The music creates a gritty, urban atmosphere, evoking the hustle and bustle of city life. The combination of the smooth vocal delivery, the driving bassline, and the rhythmic drums work together to convey a sense of determination and resilience, as if the singer is navigating through the challenges of life with a cool, collected demeanor. |