# SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot Tasks

> SwanTale unifies zero-shot and instruct-driven multi-speaker speech and audio generation in one model, achieving state-of-the-art expressiveness across monologue, dialogue, and scene-level benchmarks.

- **Source:** [arXiv](https://arxiv.org/abs/2608.02023)
- **Published:** 2026-08-05
- **Permalink:** https://picx.dev/p/GKb3xZ
- **Whiteboard:** https://picx.dev/p/GKb3xZ/image

## Summary

## Summary (Overview)

- **Unified Model for Dual Tasks**: SwanTale proposes a single model that supports both **zero-shot** (reference audio + content) and **instruct** (natural language caption) multi-speaker speech and audio generation, covering speech, environmental audio, sound effects, singing voice, and music.
- **Data Pipeline (SwanData-Caption)**: A four-stage pipeline that converts raw speech-centered media into fine-grained multi-level captions (environment, speakers, content) with targeted synthetic coverage for rare scenarios (elderly speech, short utterances, challenging pronunciation).
- **Novel Modeling Components**: SwanVAE (48 kHz, 25 Hz latent rate) with generative alignment, reward-conditioned quality control, Engram memory conditioning, and Unified MoE (task router + frame-level audio routing with dynamic Top-P) for heterogeneous acoustic modalities.
- **Training Strategy**: Curriculum learning (zero-shot → dense caption → full mixture → high-quality SFT) followed by GRPO post-training with task-specific rewards (phoneme accuracy, speaker similarity/attribute control) while preserving multi-speaker and audio capabilities.
- **State-of-the-Art Results**: SwanTale achieves best expressiveness and competitive content accuracy/quality on zero-shot monologue/dialogue, leads on instruct APS/DSD, obtains highest MOS on SwanBench-Scene, and demonstrates gains from Unified MoE in complex speech+audio generation.

## Introduction and Theoretical Foundation

Modern media creation (animation dubbing, audio drama, advertising, games, podcasts) requires both **zero-shot** synthesis (clone a voice from a reference) and **instruct** synthesis (design a voice from scratch using natural language captions). The instruct task needs to describe environment, speaker persona, local style, and audio effects; most existing systems either generate only speech or reduce speaker control to decomposed attribute labels. Three main challenges are identified:

1. **Data scarcity**: High-quality caption data requires rich audio coverage and detailed annotation; collecting expressive speech and annotating multi-level captions is expensive.
2. **Task compatibility**: Instruct samples use captions for speaker style, zero-shot samples use reference audio; joint training must preserve shared speech modeling without weakening either conditioning path.
3. **Multi-audio-modality complexity**: The waveform may contain speech, general audio, singing voice, and music – each with different temporal structures (lexical alignment, stable environments, transient effects, melodic pitch).

The paper addresses these challenges from both the data side (SwanData-Caption pipeline) and the model side (SwanTale with SwanVAE, flow-based transformer, Unified MoE, curriculum learning, GRPO).

## Methodology

### Data Pipeline: SwanData-Caption

Four stages:

- **Coverage Design**: Collects media-style data (short dramas, advertisements, animations) plus **targeted synthetic subsets** using a phoneme-aware TTS teacher: elderly speech, short utterances (1.5 s avg), challenging pronunciation (polyphonic characters, brand names). Each subset 100k utterances.
- **SwanData-Speech Preprocessing**: Vocal separation (UVR), diarization (3D-Speaker), ASR (Seed-ASR 2.0, SenseVoice), alignment (SwanAligner). Keeps segments 1–120 s, average ~10 s.
- **Caption Annotation**: Uses Seed2.0 Lite as annotator. Caption output has three fields:
  - **Environment**: scene location, recording context, persistent background sounds/effects.
  - **Speakers**: gender, age, persona, timbre, articulation, rate, accent, habitual style.
  - **Content**: chronological speech wrapped in `<S1>...</S1>` tags, audio effects in `<Audio>...</Audio>` tags, with local style descriptions.
  A **style-persona library** provides soft priors for animation, drama, and advertisement content.
- **Data Refinement**: Waveform filtering (DNSMOS, SQUIM-PESQ, STOI, SI-SDR thresholds), caption normalization (SwanVerifier checks gender/age, punctuation correction), human verification of transcription, caption quality, expressiveness (group-wise best–worst comparison). ~70M caption records.

### SwanVAE (48 kHz Latent Autoencoder)

- **Architecture**: Anti-aliased convolutional encoder, 96-dimensional latent at 25 Hz (40 ms per frame), decoder with Transformer Resampling Block (TRB) from SAME that interleaves 6 learnable output tokens per latent frame, producing 1920 samples (40 ms) per latent. Receptive field: encoder ~0.95 s, decoder end-to-end ~3.23 s.
- **Reconstruction & Adversarial Training**: Loss combines multi-resolution complex STFT, multi-band Mel, frame-wise energy loss + KL penalty (0.02). Three discriminator families: MPD, MRD, MBCSD (band-specific complex STFT discriminator up to 24 kHz). Training objective:
  $$L_{wav} = L_{rec} + \lambda_{KL} L_{KL} + \lambda_{adv} \sum_{D \in \mathcal{D}} w_D L_{adv}^D + \lambda_{fm} \sum_{D \in \mathcal{D}} L_{fm}^D$$
- **Latent Alignment Objectives** (applied to posterior mean only):
  - **Generative alignment**: Joint unconditional flow-matching predictor + causal predictor.
  - **Semantic readouts**: Multi-scale chroma, broadband energy, multi-band energy (with occasional bandwidth augmentation by downsampling).
- **Training**: 32 A100 GPUs, 100k hours of internal audio, random 3.84 s segments, 25% waveform mixing, 1% bandwidth augmentation. 407M parameters (encoder 51.7M, decoder 355M). Frozen after training.

### Flow-based Transformer

- **Conditioning Stack**:
  - **Caption branch**: Qwen-family text encoder (8B default, 32B in ablation), Engram memory layer (2/3-gram windows, gated residual update with initial negative bias).
  - **Text branch**: CosyVoice 2.0 tokenizer, lightweight Transformer, length-normalized via interpolation to audio latent timeline.
  - **Speaker-turn embeddings**: Aligned with text, derived from `<S{id}>` tags.
  - **Reward-conditioned quality control**: Quality caption (STOI, SI-SDR, PESQ, MOS levels) and quality flag \(q \in \{low, normal, high, unknown\}\) appended to global caption; at inference \(q=high\) to bias toward highest quality.
- **Flow-Matching DiT**: Non-causal, flow matching with masked MSE loss over generated region. For task \(\tau\):
  - Instruct: full caption, no prompt context. Zero-shot: content caption, prompt frames as context.
  - Training loss: $$L_{flow} = \mathbb{E}\left[ \frac{\| (1-m^{(\tau)}) \odot (\hat{v}_\theta - (x^* - \epsilon)) \|_2^2}{\max(1, \sum_i (1 - m_i^{(\tau)}))} \right]$$
- **Unified MoE**: Every second DiT feed-forward layer replaced with MoE containing:
  - **Task-shared experts** (sample-level): selected by task router, encode stable priors for instruct vs. zero-shot.
  - **Routed audio experts** (frame-level): dynamic Top-P routing, with diffusion-time-aware budget \(q(t) = \sigma(W_b e_t)\) controlling threshold \(p(t)\), null-expert bias \(b_{null}(t)\), and capacity \(c(t)\).
  - **Null experts**: skip path, reduce computation for stable regions.
  - **Training**: Gumbel noise with annealed temperature, capacity truncation, router z-loss, null-collapse penalty. Final output: \(o_{MoE} = o_{shared} + o_{audio}\).
- **Curriculum Learning**:
  1. Zero-shot base model (SwanVoice-style, 23M hours single-speaker, 1.7M hours two-speaker).
  2. Dense caption adaptation on 70M clean speech with attributes (dense FFN, 20k steps).
  3. Full caption-mixture training (10M samples, 10k steps, Unified MoE introduced).
  4. High-expressiveness SFT (1M subset, 4k steps).
- **GRPO Post-Training**:
  - **Stochastic flow policy**: Converts deterministic ODE to marginal-preserving SDE (Euler-Maruyama).
  - **Rewards**: Shared: phoneme accuracy (phone_core, phone_len), punctuation-aware pause, audio-boundary energy, waveform quality. Instruct: attribute reward (age/gender). Zero-shot: speaker similarity (WavLM-based ECAPA-TDNN).
  - **GRPO**: 8 trajectories per condition, group-relative advantage, clipped surrogate objective with KL constraint to frozen SFT reference. Anchor replay preserves multi-speaker/audio capabilities.

## Empirical Validation / Results

### SwanVAE Reconstruction

| Model | Speech (PESQ↑) | Singing (PESQ↑) | General Audio (ViSQOL↑) | Music (ViSQOL↑) |
|-------|----------------|-----------------|-------------------------|-----------------|
| DAC | 4.1178 | 3.7872 | 4.0198 | 4.1534 |
| EnCodec | 3.1872 | 2.6464 | 4.1140 | **4.2976** |
| SwanVAE (Ours) | **4.1683** | **3.9821** | **4.1269** | 4.2623 |

SwanVAE achieves best or second-best reconstruction across all domains with a single model.

### Zero-Shot TTS (SwanBench-Speech)

| Model (Monologue) | Timbre↑ | Content Error↓ | SpeechJudge↑ | Richness↑ | Hierarchy↑ |
|-------------------|---------|----------------|--------------|-----------|------------|
| SwanVoice [64] | 0.93 | 0.172 | 3.56 | 3.81 | 3.62 |
| **SwanTale (Ours)** | **0.95** | **0.086** | 3.75 | **3.90** | **3.70** |

| Model (Dialogue) | Timbre↑ | Content Error↓ | SpeechJudge↑ | Richness↑ | Hierarchy↑ |
|------------------|---------|----------------|--------------|-----------|------------|
| SwanVoice [64] | 0.92 | 0.145 | 3.70 | 3.62 | 3.71 |
| **SwanTale (Ours)** | **0.94** | **0.120** | **3.92** | **3.66** | **3.85** |

SwanTale leads on timbre consistency and expressiveness; content error and sound fidelity are competitive but not best (FishSpeech lower content error in monologue; SoulX-Podcast better in dialogue).

### Instruct TTS (InstructTTSEval)

| Model | Chinese APS↑ | Chinese DSD↑ | Chinese RP↑ | English APS↑ |
|-------|--------------|--------------|-------------|--------------|
| VoxCPM2 [118] | 85.2 | 71.5 | 60.8 | 84.2 |
| **SwanTale (Ours)** | **86.1** | 80.1 | 64.1 | **84.2** |

SwanTale ranks first on Chinese APS, ties first on English APS, second on Chinese DSD. RP is weaker; English DSD less competitive than some baselines.

### SwanBench-Scene (Mean MOS)

| Model | Overall Mean MOS↑ | Advertising | Comic Drama | General Scene |
|-------|-------------------|-------------|-------------|---------------|
| Qwen3-TTS [79] | 4.09 | 3.71 | 4.35 | 4.22 |
| Seedance 2.0 [90] | 3.86 | 3.38 | 4.35 | 4.14 |
| **SwanTale (Ours)** | **4.22** | **3.88** | **4.45** | **4.34** |

SwanTale achieves highest overall and per-scene MOS.

### SwanBench-Caption Ablation

| Setting | Instruction Accuracy↑ | Acoustic Quality↑ | Overall Expressiveness↑ |
|---------|-----------------------|--------------------|-------------------------|
| w/o MoE | 3.02 | 4.09 | 3.56 |
| SwanTale | 3.39 | 4.31 | 3.82 |
| w/ 32B CE | **3.70** | **4.34** | **3.98** |

Unified MoE improves all metrics; larger caption encoder boosts instruction accuracy significantly.

## Theoretical and Practical Implications

- **Theoretical**: SwanTale demonstrates that a single flow-based model with sparse MoE can jointly handle heterogeneous audio modalities (speech, environment, singing, music) and two conditioning paradigms (zero-shot reference vs. instruct caption). The use of generative alignment in the VAE latent space and reward-conditioned quality control (without RL) provides a framework for controllable generation without reward modeling overhead. The Engram memory layer offers a lightweight alternative to full language model integration for pattern recognition in long captions.
- **Practical**: The system enables practical media creation workflows: creators can design a voice from a persona description, reuse it via zero-shot, and generate full scenes with speech, environment, and effects in one pass. The data pipeline (SwanData-Caption) can be reused for other speech-audio tasks. The curriculum learning and GRPO recipe balances broad coverage with targeted quality improvement. Results show that expressiveness in zero-shot and instruction-following in instruct tasks can be simultaneously improved, making the model suitable for animation dubbing, audio drama, and advertising production.

## Conclusion

SwanTale unifies multi-speaker speech and audio generation for both instruct and zero-shot tasks. The SwanData-Caption pipeline provides fine-grained multi-level captions with targeted synthetic coverage. The model combines SwanVAE (high-quality 48 kHz latent representation), a flow-based transformer with reward-conditioned quality control and Engram conditioning, and Unified MoE for heterogeneous modalities. Curriculum learning and GRPO post-training progressively refine capabilities. Experiments show SwanTale leads on expressiveness in zero-shot and instruct tasks, achieves competitive content accuracy, and supports complex multi-speaker speech+audio generation.

**Future Directions**:
- Complex background music generation (type changes, emotional transitions).
- Long-form instruct generation (>2 minutes, complex multi-speaker scenes with audio effects).
- Precise local style control (continuous emotional changes, emphasis, rhythm, timing).
- Unified audio generation and editing within a common framework.

---

_Markdown view of https://picx.dev/p/GKb3xZ, served by PicX — AI-generated visual whiteboard summaries of research papers._