Summary (Overview)

• AuK is an open-source foundational model unifying speech generation and editing through a common interface of natural-language instructions and audio context, covering five task families: speech generation, content editing, enhancement/separation, paralinguistic editing, and acoustic editing.
• The training corpus comprises approximately 3.03 billion instruction–audio instances and 1.95 million hours of effective supervision across the five task families.
• The architecture combines a multimodal large language model (Qwen2.5-Omni) for semantic conditioning, a VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer with dual-stream MMDiT blocks followed by single-stream DiT blocks.
• Post-training applies human-feedback preference optimization (flow-based DPO with ordinal listwise objective) for editing and Flow-GRPO reinforcement learning for generation, followed by task-routed Decoupled DMD distillation yielding AuK-Flash (4-step inference, CFG-free, 4.5× wall-clock speedup).
• AuK achieves leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing benchmarks, while remaining competitive on signal-level restoration tasks (enhancement, separation, super-resolution).

---

Introduction and Theoretical Foundation

Recent speech generation systems have advanced from conventional text-to-speech toward zero-shot voice cloning, instruction-controlled synthesis, and flexible speech editing. However, these capabilities rarely appear in isolation in practical use. A user may need to synthesize speech in a described style, replace part of an utterance, alter emotion or accent, insert a nonverbal vocalization, isolate a speaker, or restore degraded audio.

Three key challenges motivate a unified model:

1. Differing output constraints: Generation creates new speech; content editing changes only selected regions; paralinguistic/acoustic editing must preserve linguistic content; enhancement/separation retains only scene components specified by the instruction.
2. Varying conditioning interfaces: Some tasks rely on text alone, while others require joint reasoning over an instruction and source/reference audio.
3. Heterogeneous supervision and evaluation: Recognition accuracy and speaker similarity provide scalable signals for generation, but open-ended editing depends on subjective judgments of naturalness, edit strength, contextual appropriateness, and preservation of unspecified attributes.

Theoretical foundation: AuK formulates all capabilities through a common interface—a natural-language instruction and optional audio context mapped to a target waveform. Training uses rectified flow matching with a linear interpolation between Gaussian noise and clean target latents:

$$\mathbf{z}_t = (1 - t)\mathbf{z}_0 + t\mathbf{z}_1, \qquad \mathbf{v}_t = \mathbf{z}_1 - \mathbf{z}_0 \tag{10}$$

with a masked mean-squared error objective:

$$\mathcal{L}_{\mathrm{FM}} = \frac{\|\mathbf{m} \odot (\hat{\mathbf{v}}_t - \mathbf{v}_t)\|_2^2}{\sum_i m_i} \tag{11}$$

---

Methodology

Data Construction (Five Task Families)

1. Speech Generation:

   • Zero-shot TTS: Transcript-free in-context learning. For a speaker with $n$ utterances, enumerate all $C^2_n$ unordered pairs bidirectionally, yielding $n \times (n-1)$ training instances with only prompt speech and target text (no prompt transcript).
   • Instruct TTS: Qwen3-Omni annotates utterances with free-form natural-language captions and structured attributes (gender, age, speaking rate, emotion, timbre, accent, etc.).

2. Acoustic Editing: Deterministic signal-processing transformations—five speaking-rate multipliers (0.5×–2.0×), six loudness offsets (±5–±15 dB), and six pitch shifts (±1–±3 semitones).

3. Paralinguistic Editing: Emotion editing (8 categories via Qwen3-TTS-CustomVoice + IndexTTS2), timbre editing (X-VC corpus), de-accenting (13 Chinese dialect categories via CosyVoice2 + OmniVoice), nonverbal editing (39 event types via F5-TTS reconstruction), and whisper-style conversion.

4. Content Editing: LLM-generated operator annotations for insertion/deletion/substitution; localized masked infilling with F5-TTS conditioned on masked source and target transcript.

5. Enhancement and Separation: Speech enhancement with randomized degradations (noise, reverberation, channel effects); multi-speaker separation with natural conversational mixtures; music enhancement/separation with native-song and scene-based examples.

Model Architecture

The architecture consists of three components:

• MLLM Semantic Condition: Qwen2.5-Omni encodes instruction and optional reference audio. Layer-wise hidden states are aggregated:

$$\mathbf{c}_{\mathrm{sem}} = \sum_{\ell=1}^{L} w_{\ell} \cdot \operatorname{LayerNorm}\left(\mathbf{h}^{(\ell)}\right) \tag{2}$$

• VAE Acoustic Condition: A flow-augmented audio VAE operating on 24 kHz waveforms, mapping to 64-dimensional latents at 50 Hz. The acoustic condition is:

$$\mathbf{c}_{\mathrm{ac}} = \begin{cases} \mathbf{z}_{\text{ref}}, & \text{if reference audio is available}, \\ \varnothing, & \text{otherwise}. \end{cases} \tag{4}$$

• Hybrid Transformer Backbone: 10 dual-stream MMDiT blocks followed by 20 single-stream DiT blocks (~1.5B parameters). Input streams are formed as:

$$\mathbf{s}^{(0)} = \mathcal{P}_{\mathrm{sem}}(\mathbf{c}_{\mathrm{sem}}), \qquad \mathbf{a}^{(0)} = [\mathcal{P}_{\mathrm{ref}}(\mathbf{c}_{\mathrm{ac}}); \mathcal{P}_{\mathrm{tgt}}(\mathbf{z}_t)] \tag{5}$$

Training Pipeline

1. VAE Training: 1.24M updates on ~3M hours (speech:music:audio = 6:3:1), using multi-scale log-mel loss, multi-period + multi-scale CQT discriminators, and KL regularization:

$$\mathcal{L}_{\mathrm{VAE}} = \lambda_{\mathrm{mel}}\mathcal{L}_{\mathrm{mel}} + \lambda_{\mathrm{adv}}\mathcal{L}_{\mathrm{adv}} + \lambda_{\mathrm{feat}}\mathcal{L}_{\mathrm{feat}} + \lambda_{\mathrm{KL}}\mathcal{L}_{\mathrm{KL}} \tag{9}$$

2. Unified Pre-training: Two-stage curriculum—generation-only warm-up (50k updates), then joint generation+editing (600k updates) with task mixture probabilities in Table 1, condition dropout (acoustic 0.3, unconditional 0.2), and reference augmentation (50% crop probability).

3. Editing Preference Optimization: Human feedback on 3-level ordinal scale (failure/partial/success), 818 informative groups, 9,080 rated candidates. Flow-based DPO score with ordinal listwise LiPO objective:

$$\mathcal{L}_q = \frac{1}{|\mathcal{I}_q^+||\mathcal{I}_q^-|} \sum_{i \in \mathcal{I}_q^+}\sum_{j \in \mathcal{I}_q^-} \ell(\Delta R_{ij}), \qquad \mathcal{L}_g = \frac{\sum_{q \in \mathcal{A}_g} \omega_q \mathcal{L}_q}{\sum_{q \in \mathcal{A}_g} \omega_q} \tag{12}$$

4. Generation Reinforcement Learning: Flow-GRPO with content reward (tolerant/strict error rates fused with NLL):

$$R_i^{\text{content}} = (1-\gamma)\left(1 + 4e^{-E_i^s/\tau}\right) + \gamma\left(1 + 4e^{-\mathrm{NLL}_i}\right) \tag{13}$$

speaker similarity as cosine similarity of speaker embeddings:

$$R_i^{\mathrm{spk}} = \cos\big(\psi(y_i), \psi(y_{\mathrm{ref}})\big) \tag{14}$$

and style consistency via majority-voted judge model:

$$R_i^{\text{style}} = \mathbb{1}\left[\sum_{v=1}^{V}\mathbb{1}[\operatorname{judge}_v(\mathbf{y}_i) = \text{consistent}] > V/2\right] \tag{15}$$

5. Model Acceleration: Consistency initialization (trajectory-level consistency distillation) followed by task-routed Decoupled DMD. Separation examples are routed to supervised clean-prediction regression:

$$\left(\mathcal{L}_\theta^{(i)}, \mathcal{L}_\phi^{(i)}\right) = \begin{cases} \left(\|\hat{\mathbf{z}}_{1,i} - \mathbf{z}_{1,i}\|_2^2, 0\right), & \tau_i \in \mathcal{S}, \\ \left(\mathcal{L}_{\mathrm{DMD}}^{(i)}, \mathcal{L}_{\mathrm{fake}}^{(i)}\right), & \tau_i \notin \mathcal{S}, \end{cases} \tag{16}$$

---

Empirical Validation / Results

VAE Reconstruction (Table 4)

AuK-VAE achieves best results on all four metrics across all three domains (speech, general audio, music):

Generation Ability

Zero-Shot TTS (Seed-TTS-Eval): AuK achieves the lowest average WER (2.65%) and highest average SIM (0.795), improving over Qwen3-TTS (3.07% WER) and Seed-TTS (0.778 SIM). AuK-Flash remains competitive (2.85% WER, 0.790 SIM).

Instruct TTS (InstructTTSEval): AuK achieves best Chinese DSD accuracy of 83.37%, outperforming Qwen3-TTS-VD by 2.27 points. AuK-Flash ties for best English DSD at 82.40%.

Editing Ability

MMAE-Speech: AuK achieves highest IFR (48.23%) and CR (88.11%), improving over Step-Audio-EditX by 4.71 and 10.84 points respectively. AuK-Flash achieves highest EMR (13.85%).

SpeechEditBench: AuK achieves best results in content (91.83%), emotion (9.94%), prosody (71.33%), and acoustic editing (38.50%); AuK-Flash best in paralinguistic (39.25%).

Ming-Freeform-Audio-Edit: Under the Full setting, AuK reduces average WER from 10.46%→3.09% (Chinese) and 14.28%→3.96% (English) versus Ming-UniAudio. For acoustic editing, WER reduced from 5.01%→2.02% (Chinese) and 10.75%→3.48% (English).

Enhancement and Separation

• DNS Challenge: AuK achieves lowest dWER (2.66%), both variants achieve highest SIM (0.99).
• CHiME-4: AuK-Flash achieves lowest WER (7.84%) and highest UTMOS (3.91).
• Libri2Mix: AuK achieves lowest WER (9.12%) and PER (6.63%); AuK-Flash highest OVRL (3.32) and UTMOS (4.03).
• VCTK-SR: AuK-Flash best on 6 of 7 metrics (OVRL 3.25, UTMOS 4.05, WER 2.92%); AuK highest SIM (0.97).

Key Discoveries

1. Cross-utterance in-context learning improves expressiveness and speaker similarity while enabling transcript-free cloning.
2. Emergent cross-task transfer: Whisper editing transfers to text-to-whisper generation; Chinese de-accenting transfers to English accent reduction (Indian, Japanese accents).
3. Limits of native free-form instruction following: Agent-based data pipelines increase linguistic diversity but still require the Prompt Enhancer for robust generalization.

---

Theoretical and Practical Implications

Theoretical implications:

• The unified instruction-conditioned formulation demonstrates that generation, editing, restoration, and separation can share a single objective (flow matching) and architecture, challenging the task-specific model paradigm.
• The two-stage post-training (preference optimization for subjective editing + RL for objective generation) provides a template for aligning heterogeneous capability sets with different evaluation criteria.
• Task-routed distillation reveals that uniform distribution matching can degrade certain capabilities (separation), suggesting that distillation strategies must respect task-specific distributional properties.

Practical implications:

• AuK-Flash's 4.5× speedup with 4-step CFG-free inference makes the full capability set practical for real-time applications.
• Open-source release of code and weights enables community adaptation and further research.
• The Prompt Enhancer design provides a practical bridge between free-form user requests and model capabilities, though native instruction grounding remains an open challenge.

---

Conclusion

AuK unifies speech generation and editing through a common instruction-conditioned waveform generation interface, trained on approximately 3.03 billion instruction–audio instances and 1.95 million hours of supervision across five task families. The combination of MLLM semantic conditioning, joint speech/audio/music VAE, and hybrid flow Transformer achieves leading performance on generation and editing benchmarks while remaining competitive on restoration tasks.

Future directions:

• Improving native instruction grounding and compositional generalization to reduce reliance on explicit task routing and prompt enhancement
• Scalable alignment methods for open-ended audio transformations
• Further reducing inference cost while maintaining capability breadth

The source code and model weights are released to support reproducibility and facilitate future development.