# AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing

> AuK unifies speech generation and editing across five task families via instruction-conditioned flow matching, achieving state-of-the-art results with a 4.5x faster distilled variant.

- **Source:** [arXiv](https://arxiv.org/abs/2609.08936)
- **Published:** 2026-09-10
- **Permalink:** https://picx.dev/p/xsGU3M
- **Whiteboard:** https://picx.dev/p/xsGU3M/image

## Summary

## Summary (Overview)

- **AuK** is an open-source foundational model unifying speech generation and editing through a common interface of natural-language instructions and audio context, covering five task families: speech generation, content editing, enhancement/separation, paralinguistic editing, and acoustic editing.
- The training corpus comprises approximately **3.03 billion instruction–audio instances** and **1.95 million hours** of effective supervision across the five task families.
- The architecture combines a **multimodal large language model (Qwen2.5-Omni)** for semantic conditioning, a **VAE jointly trained on speech, general audio, and music** for acoustic conditioning, and a **hybrid rectified-flow Transformer** with dual-stream MMDiT blocks followed by single-stream DiT blocks.
- Post-training applies **human-feedback preference optimization** (flow-based DPO with ordinal listwise objective) for editing and **Flow-GRPO reinforcement learning** for generation, followed by **task-routed Decoupled DMD distillation** yielding **AuK-Flash** (4-step inference, CFG-free, 4.5× wall-clock speedup).
- AuK achieves leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing benchmarks, while remaining competitive on signal-level restoration tasks (enhancement, separation, super-resolution).

---

## Introduction and Theoretical Foundation

Recent speech generation systems have advanced from conventional text-to-speech toward zero-shot voice cloning, instruction-controlled synthesis, and flexible speech editing. However, these capabilities rarely appear in isolation in practical use. A user may need to synthesize speech in a described style, replace part of an utterance, alter emotion or accent, insert a nonverbal vocalization, isolate a speaker, or restore degraded audio.

**Three key challenges motivate a unified model:**

1. **Differing output constraints**: Generation creates new speech; content editing changes only selected regions; paralinguistic/acoustic editing must preserve linguistic content; enhancement/separation retains only scene components specified by the instruction.
2. **Varying conditioning interfaces**: Some tasks rely on text alone, while others require joint reasoning over an instruction and source/reference audio.
3. **Heterogeneous supervision and evaluation**: Recognition accuracy and speaker similarity provide scalable signals for generation, but open-ended editing depends on subjective judgments of naturalness, edit strength, contextual appropriateness, and preservation of unspecified attributes.

**Theoretical foundation**: AuK formulates all capabilities through a common interface—a natural-language instruction and optional audio context mapped to a target waveform. Training uses **rectified flow matching** with a linear interpolation between Gaussian noise and clean target latents:

$$
\mathbf{z}_t = (1 - t)\mathbf{z}_0 + t\mathbf{z}_1, \qquad \mathbf{v}_t = \mathbf{z}_1 - \mathbf{z}_0 \tag{10}
$$

with a masked mean-squared error objective:

$$
\mathcal{L}_{\mathrm{FM}} = \frac{\|\mathbf{m} \odot (\hat{\mathbf{v}}_t - \mathbf{v}_t)\|_2^2}{\sum_i m_i} \tag{11}
$$

---

## Methodology

### Data Construction (Five Task Families)

1. **Speech Generation**: 
   - **Zero-shot TTS**: Transcript-free in-context learning. For a speaker with $n$ utterances, enumerate all $C^2_n$ unordered pairs bidirectionally, yielding $n \times (n-1)$ training instances with only prompt speech and target text (no prompt transcript).
   - **Instruct TTS**: Qwen3-Omni annotates utterances with free-form natural-language captions and structured attributes (gender, age, speaking rate, emotion, timbre, accent, etc.).

2. **Acoustic Editing**: Deterministic signal-processing transformations—five speaking-rate multipliers (0.5×–2.0×), six loudness offsets (±5–±15 dB), and six pitch shifts (±1–±3 semitones).

3. **Paralinguistic Editing**: Emotion editing (8 categories via Qwen3-TTS-CustomVoice + IndexTTS2), timbre editing (X-VC corpus), de-accenting (13 Chinese dialect categories via CosyVoice2 + OmniVoice), nonverbal editing (39 event types via F5-TTS reconstruction), and whisper-style conversion.

4. **Content Editing**: LLM-generated operator annotations for insertion/deletion/substitution; localized masked infilling with F5-TTS conditioned on masked source and target transcript.

5. **Enhancement and Separation**: Speech enhancement with randomized degradations (noise, reverberation, channel effects); multi-speaker separation with natural conversational mixtures; music enhancement/separation with native-song and scene-based examples.

### Model Architecture

The architecture consists of three components:

- **MLLM Semantic Condition**: Qwen2.5-Omni encodes instruction and optional reference audio. Layer-wise hidden states are aggregated:

$$
\mathbf{c}_{\mathrm{sem}} = \sum_{\ell=1}^{L} w_{\ell} \cdot \operatorname{LayerNorm}\left(\mathbf{h}^{(\ell)}\right) \tag{2}
$$

- **VAE Acoustic Condition**: A flow-augmented audio VAE operating on 24 kHz waveforms, mapping to 64-dimensional latents at 50 Hz. The acoustic condition is:

$$
\mathbf{c}_{\mathrm{ac}} = \begin{cases} \mathbf{z}_{\text{ref}}, & \text{if reference audio is available}, \\ \varnothing, & \text{otherwise}. \end{cases} \tag{4}
$$

- **Hybrid Transformer Backbone**: 10 dual-stream MMDiT blocks followed by 20 single-stream DiT blocks (~1.5B parameters). Input streams are formed as:

$$
\mathbf{s}^{(0)} = \mathcal{P}_{\mathrm{sem}}(\mathbf{c}_{\mathrm{sem}}), \qquad \mathbf{a}^{(0)} = [\mathcal{P}_{\mathrm{ref}}(\mathbf{c}_{\mathrm{ac}}); \mathcal{P}_{\mathrm{tgt}}(\mathbf{z}_t)] \tag{5}
$$

### Training Pipeline

1. **VAE Training**: 1.24M updates on ~3M hours (speech:music:audio = 6:3:1), using multi-scale log-mel loss, multi-period + multi-scale CQT discriminators, and KL regularization:

$$
\mathcal{L}_{\mathrm{VAE}} = \lambda_{\mathrm{mel}}\mathcal{L}_{\mathrm{mel}} + \lambda_{\mathrm{adv}}\mathcal{L}_{\mathrm{adv}} + \lambda_{\mathrm{feat}}\mathcal{L}_{\mathrm{feat}} + \lambda_{\mathrm{KL}}\mathcal{L}_{\mathrm{KL}} \tag{9}
$$

2. **Unified Pre-training**: Two-stage curriculum—generation-only warm-up (50k updates), then joint generation+editing (600k updates) with task mixture probabilities in Table 1, condition dropout (acoustic 0.3, unconditional 0.2), and reference augmentation (50% crop probability).

3. **Editing Preference Optimization**: Human feedback on 3-level ordinal scale (failure/partial/success), 818 informative groups, 9,080 rated candidates. Flow-based DPO score with ordinal listwise LiPO objective:

$$
\mathcal{L}_q = \frac{1}{|\mathcal{I}_q^+||\mathcal{I}_q^-|} \sum_{i \in \mathcal{I}_q^+}\sum_{j \in \mathcal{I}_q^-} \ell(\Delta R_{ij}), \qquad \mathcal{L}_g = \frac{\sum_{q \in \mathcal{A}_g} \omega_q \mathcal{L}_q}{\sum_{q \in \mathcal{A}_g} \omega_q} \tag{12}
$$

4. **Generation Reinforcement Learning**: Flow-GRPO with content reward (tolerant/strict error rates fused with NLL):

$$
R_i^{\text{content}} = (1-\gamma)\left(1 + 4e^{-E_i^s/\tau}\right) + \gamma\left(1 + 4e^{-\mathrm{NLL}_i}\right) \tag{13}
$$

speaker similarity as cosine similarity of speaker embeddings:

$$
R_i^{\mathrm{spk}} = \cos\big(\psi(y_i), \psi(y_{\mathrm{ref}})\big) \tag{14}
$$

and style consistency via majority-voted judge model:

$$
R_i^{\text{style}} = \mathbb{1}\left[\sum_{v=1}^{V}\mathbb{1}[\operatorname{judge}_v(\mathbf{y}_i) = \text{consistent}] > V/2\right] \tag{15}
$$

5. **Model Acceleration**: Consistency initialization (trajectory-level consistency distillation) followed by task-routed Decoupled DMD. Separation examples are routed to supervised clean-prediction regression:

$$
\left(\mathcal{L}_\theta^{(i)}, \mathcal{L}_\phi^{(i)}\right) = \begin{cases} \left(\|\hat{\mathbf{z}}_{1,i} - \mathbf{z}_{1,i}\|_2^2, 0\right), & \tau_i \in \mathcal{S}, \\ \left(\mathcal{L}_{\mathrm{DMD}}^{(i)}, \mathcal{L}_{\mathrm{fake}}^{(i)}\right), & \tau_i \notin \mathcal{S}, \end{cases} \tag{16}
$$

---

## Empirical Validation / Results

### VAE Reconstruction (Table 4)

AuK-VAE achieves **best results on all four metrics across all three domains** (speech, general audio, music):

| Domain | Method | PESQ↑ | STOI↑ | Mel Dist↓ | STFT↓ |
|--------|--------|-------|-------|-----------|-------|
| Speech | MiniMax-H3-AudioVAE | 3.633 | 0.968 | 0.695 | 1.615 |
| Speech | **AuK-VAE** | **4.143** | **0.982** | **0.574** | **1.457** |
| Audio | **AuK-VAE** | **3.833** | **0.900** | **0.571** | **1.842** |
| Music | **AuK-VAE** | **3.884** | **0.927** | **0.525** | **1.721** |

### Generation Ability

**Zero-Shot TTS (Seed-TTS-Eval)**: AuK achieves the lowest average WER (2.65%) and highest average SIM (0.795), improving over Qwen3-TTS (3.07% WER) and Seed-TTS (0.778 SIM). AuK-Flash remains competitive (2.85% WER, 0.790 SIM).

**Instruct TTS (InstructTTSEval)**: AuK achieves best Chinese DSD accuracy of 83.37%, outperforming Qwen3-TTS-VD by 2.27 points. AuK-Flash ties for best English DSD at 82.40%.

### Editing Ability

**MMAE-Speech**: AuK achieves highest IFR (48.23%) and CR (88.11%), improving over Step-Audio-EditX by 4.71 and 10.84 points respectively. AuK-Flash achieves highest EMR (13.85%).

**SpeechEditBench**: AuK achieves best results in content (91.83%), emotion (9.94%), prosody (71.33%), and acoustic editing (38.50%); AuK-Flash best in paralinguistic (39.25%).

**Ming-Freeform-Audio-Edit**: Under the Full setting, AuK reduces average WER from 10.46%→3.09% (Chinese) and 14.28%→3.96% (English) versus Ming-UniAudio. For acoustic editing, WER reduced from 5.01%→2.02% (Chinese) and 10.75%→3.48% (English).

### Enhancement and Separation

- **DNS Challenge**: AuK achieves lowest dWER (2.66%), both variants achieve highest SIM (0.99).
- **CHiME-4**: AuK-Flash achieves lowest WER (7.84%) and highest UTMOS (3.91).
- **Libri2Mix**: AuK achieves lowest WER (9.12%) and PER (6.63%); AuK-Flash highest OVRL (3.32) and UTMOS (4.03).
- **VCTK-SR**: AuK-Flash best on 6 of 7 metrics (OVRL 3.25, UTMOS 4.05, WER 2.92%); AuK highest SIM (0.97).

### Key Discoveries

1. **Cross-utterance in-context learning** improves expressiveness and speaker similarity while enabling transcript-free cloning.
2. **Emergent cross-task transfer**: Whisper editing transfers to text-to-whisper generation; Chinese de-accenting transfers to English accent reduction (Indian, Japanese accents).
3. **Limits of native free-form instruction following**: Agent-based data pipelines increase linguistic diversity but still require the Prompt Enhancer for robust generalization.

---

## Theoretical and Practical Implications

**Theoretical implications:**

- The unified instruction-conditioned formulation demonstrates that generation, editing, restoration, and separation can share a single objective (flow matching) and architecture, challenging the task-specific model paradigm.
- The two-stage post-training (preference optimization for subjective editing + RL for objective generation) provides a template for aligning heterogeneous capability sets with different evaluation criteria.
- Task-routed distillation reveals that uniform distribution matching can degrade certain capabilities (separation), suggesting that distillation strategies must respect task-specific distributional properties.

**Practical implications:**

- AuK-Flash's 4.5× speedup with 4-step CFG-free inference makes the full capability set practical for real-time applications.
- Open-source release of code and weights enables community adaptation and further research.
- The Prompt Enhancer design provides a practical bridge between free-form user requests and model capabilities, though native instruction grounding remains an open challenge.

---

## Conclusion

AuK unifies speech generation and editing through a common instruction-conditioned waveform generation interface, trained on approximately 3.03 billion instruction–audio instances and 1.95 million hours of supervision across five task families. The combination of MLLM semantic conditioning, joint speech/audio/music VAE, and hybrid flow Transformer achieves leading performance on generation and editing benchmarks while remaining competitive on restoration tasks.

**Future directions:**
- Improving native instruction grounding and compositional generalization to reduce reliance on explicit task routing and prompt enhancement
- Scalable alignment methods for open-ended audio transformations
- Further reducing inference cost while maintaining capability breadth

The source code and model weights are released to support reproducibility and facilitate future development.

---

_Markdown view of https://picx.dev/p/xsGU3M, served by PicX — AI-generated visual whiteboard summaries of research papers._