How AI Music Generation Works

// In plain English

AI music generation is software that takes a short text description — "upbeat 80s synth-pop, catchy chorus, lyrics about a road trip" — and produces a complete, listenable song: melody, harmony, rhythm, instruments, and singing voice included. No musicians, no studio, no sheet music. You describe what you want, the model composes, arranges, and performs it, all in one pass.

The headline tools are Suno and Udio, both launched in 2024 and now used by tens of millions of people. Meta's open-source MusicGen sits at the other end of the spectrum — a research model you can run yourself that generates instrumental music from a text or melody prompt. Each takes a different technical approach, but all three share the same core idea: learn the statistical patterns of music by training on huge libraries of songs, then generate new audio that fits a given description.

A useful intuition: these are not search engines that retrieve existing songs. They are composition engines that have absorbed the grammar of music — what chord tends to follow another, how a verse differs from a chorus, what a snare drum sounds like in a trap beat versus a jazz standard — and use that grammar to write something new. Every output is, in principle, a song that has never existed before.

// Why it matters

Music licensing has always been a pain point. Sync licenses for commercial videos, background music for podcasts, custom jingles for apps — all expensive, slow, and full of rights friction. AI music generation collapses the time and cost to near zero for a first draft, which changes the economics for indie filmmakers, game developers, content creators, and anyone building a product that needs audio.

On the creative side, the tools lower the floor for non-musicians. Someone who can describe music in words but cannot play an instrument or write notation can now hear their ideas. That is a genuine expansion of who gets to make music, even if what comes out still has rough edges.

The stakes are high enough that the major record labels — Sony, Universal, and Warner — sued both Suno and Udio in June 2024 for alleged copyright infringement in how they trained their models. Universal settled with Udio in October 2025, creating the first major-label licensing template for AI music; Suno is still fighting the case on fair-use grounds as of mid-2026. The outcome will shape who can build these tools and on what terms.

Market size matters too: the generative AI music market grew from roughly $570 million in 2024 to an estimated $1.98 billion in 2026. Suno alone reported 2 million paid subscribers and $300 million in annual recurring revenue. This is no longer a research curiosity — it is a commercial industry.

// How it works

The core challenge in AI music generation is representation: raw audio is a waveform — millions of numbers per second — far too dense for a language model to process directly. Every major music AI system therefore starts by compressing audio into a compact, discrete form called tokens, processes those tokens with a transformer, then decodes the output back into a waveform. This loop — encode, predict, decode — is the skeleton underneath Suno, Udio, and MusicGen alike.

Step 1: Audio tokenization with EnCodec

Meta's EnCodec (used directly in MusicGen and influencing the broader field) is a neural audio codec that compresses a 32 kHz audio waveform into a small grid of discrete codes using a technique called residual vector quantization (RVQ). Think of it as MP3 compression but learned end-to-end: four separate "codebooks" each capture a different layer of the audio signal — coarse pitch and rhythm in the first, finer timbral detail in each subsequent one. A second or so of music that would be 32,000 raw samples becomes just 50 sets of four small integers — a 600x compression ratio. The decoder is a convolutional network that reconstructs a waveform from those codes that is perceptually close to the original.

Step 2: Autoregressive transformer generation

Once audio is tokens, generation works like a language model. The system receives a text prompt (encoded by a text model such as T5), and a transformer predicts the next token in the sequence — then the next, then the next — until the full song is built up token by token. MusicGen handles all four RVQ codebooks in a single pass using a clever interleaving trick: it introduces a small delay between codebooks so the model can predict them in parallel rather than one at a time, reducing the number of autoregressive steps needed per second of audio. Suno and Udio use proprietary but broadly similar architectures.

Step 3: Lyrics-to-song and vocal synthesis

Consumer tools like Suno go further: they accept lyrics and generate a singing voice. This requires the model to learn the relationship between text syllables, melody, timing, and phoneme-level vocal production — essentially a text-to-speech problem wrapped inside a music generation problem. The system must match syllable stress to melodic contour, decide where a word falls on the beat, and produce a voice timbre that fits the requested genre. Suno v5.5 (March 2026) introduced voice capture: users can record themselves singing a few bars, and the model incorporates that vocal identity into generated tracks — a form of style transfer applied specifically to the singing voice.

// Suno, Udio, and MusicGen compared

The three models represent three distinct points on the access and capability spectrum.

Suno is optimized for full song production from a single prompt. You can write lyrics, pick a style, and receive a complete track with a singing voice in under a minute. Version 4 (Nov 2024) and v4.5 (May 2025) pushed output length to 8 minutes and improved vocal expressiveness; v5 added 44.1 kHz output quality; v5.5 added voice capture.

Udio targets a slightly more production-oriented user. Its headline differentiator is inpainting: you can select any 2-second window in a generated track and describe a change — "add a trumpet here" or "make the drums drop out" — and Udio regenerates only that segment while leaving the rest intact. No other major consumer music AI offered this at launch. Udio v1.5 (mid-2025) added key guidance (specify the musical key) and support for lyrics in dozens of languages.

MusicGen is Meta's open-source research contribution, available on Hugging Face under the MIT license. It does not produce vocals; it generates instrumental music from a text description or a melody reference. The practical value is control and transparency: you can run it locally, fine-tune on your own dataset, and inspect every part of the pipeline. The model comes in 300M, 1.5B, and 3.3B parameter sizes.

// Copyright, training data, and the legal frontier

The central legal question is whether training an AI model on copyrighted recordings constitutes copyright infringement. In June 2024, Sony Music, Universal Music Group, and Warner Records filed suits against Suno and Udio in US federal courts, alleging that the companies trained their models on copyrighted recordings without a license. Both companies acknowledged training on copyrighted material but argued the use is fair use — that learning patterns from recordings is transformative and does not reproduce them.

Suno's legal argument centers on the idea that its model does not store or replay any original recording: it learns abstract musical patterns — chord progressions, rhythmic structures, timbral characteristics — and generates entirely new audio. Critics counter that if an AI can reproduce the distinctive sound of a specific artist on demand, the practical effect on the market is the same as infringement, regardless of how it's generated.

By late 2025, Universal Music settled with Udio, establishing the first major-label licensing framework for AI music generation. Warner settled with Suno. Sony continued litigating against both. Independent musicians filed separate class-action suits arguing that major-label settlements do not protect smaller rights holders. Suno's fair-use case has a summary judgment hearing scheduled for July 2026.

For users, the practical copyright question is different: who owns the output? In the US, the Copyright Office has consistently held that purely AI-generated works (with no human creative selection) are not copyrightable. If you write specific lyrics, arrange sections, and curate outputs, you may have a protectable work in the human-authored elements. Platforms differ: Suno grants commercial licenses to paying subscribers for tracks they generate, but retains the right to use those tracks to train future models.

// Going deeper

Residual vector quantization is the key enabler. The reason modern music AI works at all is that EnCodec and its successors found a way to compress audio into a sequence of discrete tokens without losing the perceptual qualities that make music sound good. RVQ is a nested approximation: the first codebook captures the coarsest structure; each subsequent codebook learns to encode the residual error left by the previous one. More codebooks = higher fidelity, but also more tokens per second for the transformer to predict. The MusicGen team's interleaving trick (predicting codebooks in parallel with a small delay offset) was the engineering insight that made the model tractable.

Diffusion in the audio domain. Autoregressive transformers are not the only game in town. Stable Audio Open (Stability AI, open weights) uses a latent diffusion approach: it compresses audio with a VAE, denoises in that latent space with a U-Net, and conditions on text and timing metadata. The advantage is that diffusion models can generate the whole clip in one shot rather than token-by-token, which can be faster and produces different quality trade-offs. The same fundamental tension you see in image generation — autoregressive coherence vs. diffusion fidelity — plays out in audio too.

Style transfer vs. fine-tuning. Udio's inpainting and Suno's voice capture are forms of inference-time style control — you guide the model at generation time without retraining it. A more powerful (and more technically demanding) approach is fine-tuning: training the model on a small dataset of target-style music so that its default outputs drift toward that style. This is the same technique used to customize language models (see what is fine-tuning and LoRA), and it applies just as cleanly to MusicGen — load the pretrained weights, run a few thousand steps on your target genre, and the model's musical vocabulary shifts.

The structure problem. The hardest thing today's music AI gets wrong is large-scale structure: verse-chorus-bridge architecture, key changes, dynamic builds and drops that feel intentional, a hook that recurs with variation. Token-by-token prediction excels at local coherence (the next beat sounds right given the last bar) but struggles with the macro-level decisions a human composer makes when laying out a song. This is analogous to the lost-in-the-middle problem in LLMs — the model loses the thread over long ranges. Systems that add explicit structural conditioning ("this is bar 1 of the verse") show progress, but it remains the frontier.