Summary (Overview)

• Unified Model for Dual Tasks: SwanTale proposes a single model that supports both zero-shot (reference audio + content) and instruct (natural language caption) multi-speaker speech and audio generation, covering speech, environmental audio, sound effects, singing voice, and music.
• Data Pipeline (SwanData-Caption): A four-stage pipeline that converts raw speech-centered media into fine-grained multi-level captions (environment, speakers, content) with targeted synthetic coverage for rare scenarios (elderly speech, short utterances, challenging pronunciation).
• Novel Modeling Components: SwanVAE (48 kHz, 25 Hz latent rate) with generative alignment, reward-conditioned quality control, Engram memory conditioning, and Unified MoE (task router + frame-level audio routing with dynamic Top-P) for heterogeneous acoustic modalities.
• Training Strategy: Curriculum learning (zero-shot → dense caption → full mixture → high-quality SFT) followed by GRPO post-training with task-specific rewards (phoneme accuracy, speaker similarity/attribute control) while preserving multi-speaker and audio capabilities.
• State-of-the-Art Results: SwanTale achieves best expressiveness and competitive content accuracy/quality on zero-shot monologue/dialogue, leads on instruct APS/DSD, obtains highest MOS on SwanBench-Scene, and demonstrates gains from Unified MoE in complex speech+audio generation.

Introduction and Theoretical Foundation

Modern media creation (animation dubbing, audio drama, advertising, games, podcasts) requires both zero-shot synthesis (clone a voice from a reference) and instruct synthesis (design a voice from scratch using natural language captions). The instruct task needs to describe environment, speaker persona, local style, and audio effects; most existing systems either generate only speech or reduce speaker control to decomposed attribute labels. Three main challenges are identified:

1. Data scarcity: High-quality caption data requires rich audio coverage and detailed annotation; collecting expressive speech and annotating multi-level captions is expensive.
2. Task compatibility: Instruct samples use captions for speaker style, zero-shot samples use reference audio; joint training must preserve shared speech modeling without weakening either conditioning path.
3. Multi-audio-modality complexity: The waveform may contain speech, general audio, singing voice, and music – each with different temporal structures (lexical alignment, stable environments, transient effects, melodic pitch).

The paper addresses these challenges from both the data side (SwanData-Caption pipeline) and the model side (SwanTale with SwanVAE, flow-based transformer, Unified MoE, curriculum learning, GRPO).

Methodology

Data Pipeline: SwanData-Caption

Four stages:

• Coverage Design: Collects media-style data (short dramas, advertisements, animations) plus targeted synthetic subsets using a phoneme-aware TTS teacher: elderly speech, short utterances (1.5 s avg), challenging pronunciation (polyphonic characters, brand names). Each subset 100k utterances.
• SwanData-Speech Preprocessing: Vocal separation (UVR), diarization (3D-Speaker), ASR (Seed-ASR 2.0, SenseVoice), alignment (SwanAligner). Keeps segments 1–120 s, average ~10 s.
• Caption Annotation: Uses Seed2.0 Lite as annotator. Caption output has three fields:
  • Environment: scene location, recording context, persistent background sounds/effects.
  • Speakers: gender, age, persona, timbre, articulation, rate, accent, habitual style.
  • Content: chronological speech wrapped in <S1>...</S1> tags, audio effects in <Audio>...</Audio> tags, with local style descriptions. A style-persona library provides soft priors for animation, drama, and advertisement content.
• Data Refinement: Waveform filtering (DNSMOS, SQUIM-PESQ, STOI, SI-SDR thresholds), caption normalization (SwanVerifier checks gender/age, punctuation correction), human verification of transcription, caption quality, expressiveness (group-wise best–worst comparison). ~70M caption records.

SwanVAE (48 kHz Latent Autoencoder)

• Architecture: Anti-aliased convolutional encoder, 96-dimensional latent at 25 Hz (40 ms per frame), decoder with Transformer Resampling Block (TRB) from SAME that interleaves 6 learnable output tokens per latent frame, producing 1920 samples (40 ms) per latent. Receptive field: encoder ~0.95 s, decoder end-to-end ~3.23 s.
• Reconstruction & Adversarial Training: Loss combines multi-resolution complex STFT, multi-band Mel, frame-wise energy loss + KL penalty (0.02). Three discriminator families: MPD, MRD, MBCSD (band-specific complex STFT discriminator up to 24 kHz). Training objective: $$L_{wav} = L_{rec} + \lambda_{KL} L_{KL} + \lambda_{adv} \sum_{D \in \mathcal{D}} w_D L_{adv}^D + \lambda_{fm} \sum_{D \in \mathcal{D}} L_{fm}^D$$
• Latent Alignment Objectives (applied to posterior mean only):
  • Generative alignment: Joint unconditional flow-matching predictor + causal predictor.
  • Semantic readouts: Multi-scale chroma, broadband energy, multi-band energy (with occasional bandwidth augmentation by downsampling).
• Training: 32 A100 GPUs, 100k hours of internal audio, random 3.84 s segments, 25% waveform mixing, 1% bandwidth augmentation. 407M parameters (encoder 51.7M, decoder 355M). Frozen after training.

Flow-based Transformer

• Conditioning Stack:
  • Caption branch: Qwen-family text encoder (8B default, 32B in ablation), Engram memory layer (2/3-gram windows, gated residual update with initial negative bias).
  • Text branch: CosyVoice 2.0 tokenizer, lightweight Transformer, length-normalized via interpolation to audio latent timeline.
  • Speaker-turn embeddings: Aligned with text, derived from <S{id}> tags.
  • Reward-conditioned quality control: Quality caption (STOI, SI-SDR, PESQ, MOS levels) and quality flag (q \in {low, normal, high, unknown}) appended to global caption; at inference (q=high) to bias toward highest quality.
• Flow-Matching DiT: Non-causal, flow matching with masked MSE loss over generated region. For task (\tau):
  • Instruct: full caption, no prompt context. Zero-shot: content caption, prompt frames as context.
  • Training loss: $L_{flow} = \mathbb{E}\left[ \frac{\| (1-m^{(\tau)}) \odot (\hat{v}_\theta - (x^* - \epsilon)) \|_2^2}{\max(1, \sum_i (1 - m_i^{(\tau)}))} \right]$
• Unified MoE: Every second DiT feed-forward layer replaced with MoE containing:
  • Task-shared experts (sample-level): selected by task router, encode stable priors for instruct vs. zero-shot.
  • Routed audio experts (frame-level): dynamic Top-P routing, with diffusion-time-aware budget (q(t) = \sigma(W_b e_t)) controlling threshold (p(t)), null-expert bias (b_{null}(t)), and capacity (c(t)).
  • Null experts: skip path, reduce computation for stable regions.
  • Training: Gumbel noise with annealed temperature, capacity truncation, router z-loss, null-collapse penalty. Final output: (o_{MoE} = o_{shared} + o_{audio}).
• Curriculum Learning:
  1. Zero-shot base model (SwanVoice-style, 23M hours single-speaker, 1.7M hours two-speaker).
  2. Dense caption adaptation on 70M clean speech with attributes (dense FFN, 20k steps).
  3. Full caption-mixture training (10M samples, 10k steps, Unified MoE introduced).
  4. High-expressiveness SFT (1M subset, 4k steps).
• GRPO Post-Training:
  • Stochastic flow policy: Converts deterministic ODE to marginal-preserving SDE (Euler-Maruyama).
  • Rewards: Shared: phoneme accuracy (phone_core, phone_len), punctuation-aware pause, audio-boundary energy, waveform quality. Instruct: attribute reward (age/gender). Zero-shot: speaker similarity (WavLM-based ECAPA-TDNN).
  • GRPO: 8 trajectories per condition, group-relative advantage, clipped surrogate objective with KL constraint to frozen SFT reference. Anchor replay preserves multi-speaker/audio capabilities.

Empirical Validation / Results

SwanVAE Reconstruction

SwanVAE achieves best or second-best reconstruction across all domains with a single model.

Zero-Shot TTS (SwanBench-Speech)

SwanTale leads on timbre consistency and expressiveness; content error and sound fidelity are competitive but not best (FishSpeech lower content error in monologue; SoulX-Podcast better in dialogue).

Instruct TTS (InstructTTSEval)

SwanTale ranks first on Chinese APS, ties first on English APS, second on Chinese DSD. RP is weaker; English DSD less competitive than some baselines.

SwanBench-Scene (Mean MOS)

SwanTale achieves highest overall and per-scene MOS.

SwanBench-Caption Ablation

Unified MoE improves all metrics; larger caption encoder boosts instruction accuracy significantly.

Theoretical and Practical Implications

• Theoretical: SwanTale demonstrates that a single flow-based model with sparse MoE can jointly handle heterogeneous audio modalities (speech, environment, singing, music) and two conditioning paradigms (zero-shot reference vs. instruct caption). The use of generative alignment in the VAE latent space and reward-conditioned quality control (without RL) provides a framework for controllable generation without reward modeling overhead. The Engram memory layer offers a lightweight alternative to full language model integration for pattern recognition in long captions.
• Practical: The system enables practical media creation workflows: creators can design a voice from a persona description, reuse it via zero-shot, and generate full scenes with speech, environment, and effects in one pass. The data pipeline (SwanData-Caption) can be reused for other speech-audio tasks. The curriculum learning and GRPO recipe balances broad coverage with targeted quality improvement. Results show that expressiveness in zero-shot and instruction-following in instruct tasks can be simultaneously improved, making the model suitable for animation dubbing, audio drama, and advertising production.

Conclusion

SwanTale unifies multi-speaker speech and audio generation for both instruct and zero-shot tasks. The SwanData-Caption pipeline provides fine-grained multi-level captions with targeted synthetic coverage. The model combines SwanVAE (high-quality 48 kHz latent representation), a flow-based transformer with reward-conditioned quality control and Engram conditioning, and Unified MoE for heterogeneous modalities. Curriculum learning and GRPO post-training progressively refine capabilities. Experiments show SwanTale leads on expressiveness in zero-shot and instruct tasks, achieves competitive content accuracy, and supports complex multi-speaker speech+audio generation.

Future Directions:

• Complex background music generation (type changes, emotional transitions).
• Long-form instruct generation (>2 minutes, complex multi-speaker scenes with audio effects).
• Precise local style control (continuous emotional changes, emphasis, rhythm, timing).
• Unified audio generation and editing within a common framework.