YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality

Summary (Overview)

• Unified Architecture: YuE2 introduces a single AR–NAR Mixture-of-Transformers (MoT) model that first generates a readable symbolic score (melody, harmony, rhythm, form), then expands it into semantic music tokens, and finally realizes it as full-song audio—unifying symbolic and audio music generation in one checkpoint.

• Symbolic Planning Improves Quality: Expert listening tests show that symbolic planning improves perceived song quality, with 49.3% of overall preferences favoring planning versus 34.6% without planning, demonstrating that explicit composition before audio generation enhances musicality.

• Frontier Performance: On WildSongBench, YuE2 scores 6.73 on SongBench Global Avg, exceeding all evaluated public baselines; best-of-8 selection reaches 6.96, the highest observed mean among all evaluated systems including proprietary ones.

• New Supervision Methods: The paper introduces MERT2 (music representation learning) and SheetSage2 (lead-sheet transcription) to construct symbolic and semantic supervision from recordings, with MERT2 setting state-of-the-art results on 14 of 15 MARBLE metrics.

• Versatile Score Interface: The same checkpoint supports creation, controlled editing, zero-shot cover generation (surpassing dedicated cover systems), and agentic music editing through external language models.

Introduction and Theoretical Foundation

Progressive Musical Commitment

The paper frames music generation through a hierarchy of representations organized by the decisions they settle:

• Text and lyrics state intent
• Score commits to melody, harmony, rhythm, and form
• Performance and audio representations resolve timing, articulation, timbre, expression, and production

This framework, called progressive musical commitment, suggests each stage settles additional decisions while leaving remaining choices to lower stages. The space between a score and its sound is room for interpretation.

The Gap in Existing Systems

The authors identify that existing generators expose only one side of this hierarchy:

• Symbolic models make composition readable but stop before finished recordings
• Audio models produce complete songs but leave composition implicit

Core Research Question

Can a single model traverse this hierarchy, making composition readable and editable while improving the quality of the finished song?

Key Contributions

1. YuE2: Unifies symbolic and audio generation through symbolic planning, reaching frontier full-song quality with readable scores enabling editing, covers, and agentic editing
2. Symbolic planning validation: Shows that composing a score first improves perceived song quality
3. MERT2 and SheetSage2: Supply semantic and symbolic supervision from recordings

Methodology

Generation Framework

The model factorizes generation as:

$$p_{\theta}(s, c, z \mid y) = p_{\theta, \mathrm{AR}}(s, c \mid y) p_{\theta, \mathrm{NAR}}(z \mid y, s, c)$$

where:

• $s$ = symbolic composition (ABC notation)
• $c$ = semantic tokens (25 Hz)
• $z$ = continuous acoustic latents (25 Hz, 64-dimensional)
• $y$ = text conditions and lyrics

AR–NAR Mixture-of-Transformers

The model uses a 28-layer backbone with separate AR and NAR normalization, projections, and MLP experts per layer, sharing one attention computation. The hybrid attention mask is:

• AR stream: Predicts symbolic and semantic tokens causally
• NAR stream: Predicts flow velocity over acoustic latents with bidirectional latent attention

Training Objectives

Autoregressive loss (next-token cross-entropy):

$$\mathcal{L}_{\mathrm{AR}} = -\frac{1}{|\mathcal{A}|} \sum_{i \in \mathcal{A}} \log p_{\theta}(w_i \mid w_{<i})$$

Flow matching loss (conditional flow matching):

$$\mathcal{L}_{\mathrm{FM}} = \frac{1}{64|\mathcal{Z}|} \sum_{i \in \mathcal{Z}} \| v_{\theta}(z_t, t, y, s, c)_i - (\epsilon_i - z_{0,i}) \|_2^2$$

Joint objective:

$$\mathcal{L} = 0.25 \mathcal{L}_{\mathrm{AR}} + \mathcal{L}_{\mathrm{FM}}$$

Training Data and Tasks

• ~346,000 hours of music training data
• Songs packed into 24,576-position contexts without splitting
• Four training tasks: with/without score, with/without semantic tokens
• 3.58B parameters total
• Text and lyric conditioning can be dropped for classifier-free guidance

MERT2: Multi-View Target Synthesis

MERT2 uses a four-stage curriculum:

1. Foundation Pretraining: Predict four code streams from masked audio using a ConvNeXt–Conformer encoder
2. Causal Adaptation: Switch to causal attention for tokenizer branch
3. Supervised Fine-Tuning: Add lyric CTC, mel, and chroma reconstruction objectives
4. Semantic Quantization: Insert a 32,768-entry clustered vector quantizer between layers 13 and 14

The multi-view target synthesis combines frozen MuQ and Qwen2-Audio-Instruct encoders through a shared bottleneck with four-level residual vector quantization.

SheetSage2: Full-Song Transcription

• SheetSage2-Prober: Non-autoregressive model trained on human-annotated datasets and MIDI-rendered audio with task-specific CRF-based structured decoding
• SheetSage2-AR: Six-layer autoregressive RoFormer decoder with full-context MERT2-FS encoder, generating chronological event sequences converted to ABC notation

Empirical Validation / Results

Frontier Full-Song Quality (WildSongBench)

Table 1: Comparison with proprietary systems on WSB (192 prompts)

Table 2: Comparison with publicly released systems on WSB

Expert Listening Results

• YuE2 (best-of-8) vs Suno v4.5: 57.3% favor YuE2 vs 30.5% for Suno v4.5
• YuE2 (best-of-8) vs Suno v5: Nearly balanced at 40.4% vs 39.9%
• Audio quality: Average tie-adjusted preference of 58.9% across six proprietary baselines (CR1 95% CI 55.6–62.2%)

Symbolic Planning Ablation

• Overall quality: 49.3% favor planning vs 34.6% without (p = 0.0070)
• Musicality: 45.0% vs 29.4% (p = 0.0141)
• Melody: 44.0% vs 29.5%
• Chord progression: 39.0% vs 21.0%

Unified MoT vs Separate LM+DiT

• Overall quality: 53.4% favor unified MoT vs 35.6% (p = 0.0084)
• Audio quality: 48.5% vs 29.6% (p = 3.06 × 10⁻⁴)

Score–Audio Agreement

Table 3: Score–audio consistency on WSB

Score Editing Results

Table 4: Score editing on WildSongBench (0–100 scale)

(a) Edit adherence

(b) Content preservation

Zero-Shot Cover Generation

Table 5: YuE2 with full scores leads both evaluated cover systems on all eight retrieval measures

MERT2 Music Understanding Results

Table 6: MERT2-30s leads on 14 of 15 MARBLE metrics

Key results (MERT2-30s vs best prior baseline):

• MTT Tagging ROC: 91.91 vs 91.7 (PupuJEPA-Large)
• GTZAN Genre: 91.72 vs 86.9
• EmoMusic $R_V^2$: 63.23 vs 62.5
• EmoMusic $R_A^2$: 80.01 vs 78.5
• MTG-Jamendo Instrument ROC: 80.27 vs 78.4

Semantic Tokenizer Evaluation (375 bits/s)

Table 7: MERT2 tokenizer leads five of six metrics among quantized representations

SheetSage2-AR Transcription Results

Table 8: SheetSage2-AR achieves highest scores on 12 of 15 benchmark–metric pairs

Notable improvements:

• RWC-Pop vocal melody F1: 82.51 vs 62.71 (SheetSage1), +19.80 points
• Rock Corpus vocal melody F1: 67.08 vs 49.19, +17.89 points
• JAAH chord maj/min: 64.50% vs 59.45 (ChordFormer), +5.06 points

Theoretical and Practical Implications

Theoretical Significance

1. Progressive Musical Commitment validated: The paper provides empirical evidence that generating through hierarchical representations (score → semantic tokens → audio) improves perceived quality, supporting the theoretical framework that explicit composition aids generation.

2. Unified vs. separate architectures: The MoT architecture demonstrates that jointly learning semantic-token prediction and acoustic generation outperforms the widely used LM+DiT design, suggesting benefits from shared representations and attention.

3. Multi-View Target Synthesis: The Platonic Representation Hypothesis motivates the approach of constraining discrete targets by multiple encoder views, showing that complementary representations can converge toward shared musical structure.

Practical Implications

1. Editable music generation: The readable score interface enables:

   • Local edits (melody, harmony, rhythm, key, tempo) with high adherence and preservation
   • Zero-shot covers without cover-specific training
   • Agentic editing through external language models

2. Scalable supervision: MERT2 and SheetSage2 provide a pipeline for constructing aligned symbolic and semantic supervision from unlabeled recordings, addressing the lack of aligned score-audio data.

3. Frontier quality at smaller scale: YuE2 achieves competitive results with proprietary systems at 3.58B parameters, fewer than six of eight public baselines.

4. Trade-off insights: The cover generation results reveal a tension between work-identity preservation and style adaptation, with harmony playing a crucial role in musical style.

Conclusion

YuE2 unifies readable composition and full-song audio generation at frontier quality through symbolic planning. The single AR–NAR Mixture-of-Transformers model first writes an editable score, then expands it into semantic tokens and acoustic latents. Expert listening confirms that symbolic planning improves perceived quality, and the unified architecture outperforms separate LM+DiT designs.

The model achieves state-of-the-art results on WildSongBench, competitive performance with proprietary systems, and enables novel capabilities including controlled editing, zero-shot covers, and agentic music editing—all with one checkpoint. MERT2 and SheetSage2 provide the semantic and symbolic supervision that makes this hierarchy learnable from recordings at scale.

Future directions suggested by the work include:

• Further exploration of the trade-off between score fidelity and style adaptation
• Scaling the approach to more diverse musical genres and styles
• Extending agentic editing capabilities for more complex compositional tasks