Music & Sound Generation
Generate a piece of music or a sound effect from a description
WHAT THIS CAPABILITY MEANS
Takes a text description, optionally with lyrics or a style reference, and outputs audio — a full song, an ambient bed or a sound effect. It must handle melody, harmony, arrangement and timbre at once, adding a layer of musical structure on top of speech synthesis. Unlike text-to-speech the output is not language but rhythmically and tonally organised audio.
How it is done
The mainstream uses an audio latent representation with diffusion or autoregressive generation: audio is first compressed into discrete or continuous tokens, generated under conditioning, and decoded back to a waveform. Structural tags during training teach the model the order of intro, verse and chorus. Lyrics and vocals are produced by separate alignment and synthesis modules and then mixed with the accompaniment into a finished track.
Representative products
3Suno
2023Generates complete songs with vocals from a single description
Lyria
2023Generates instrumental and vocal music from text prompts
ElevenLabs
2022A multilingual text-to-speech service with natural, cloneable voices
Organizations involved
Typical uses
- Background music for short video and podcasts
- Sound effects for games and apps
- Scores for ads and promotional films
- Creative demos and arrangement ideas
How it is evaluated
- FAD
- Distribution distance between generated and real music; lower is better
- MOS
- Mean opinion score for listening quality
- Human preference
- Pairwise judgement of melodic and arrangement appeal
Limits and hard parts
- Long-form structure collapses: chorus returns and arrangement layers fail to hold together
- Lyrics and melody misalign and enunciation blurs, with mispronounced or swallowed syllables
- Imitating the style of a living artist under copyright carries risk and needs checking before commercial use
Concepts behind it
Diffusion Models
Learn a thousand tiny denoising steps, and you can build an image from pure noise
Generative Models: An Overview
Discriminative models answer "what is this"; generative models answer "what should this look like"
Autoencoders & VAE
Squeeze information through a bottleneck, then let it grow back