Summary (Overview)

• SwanVoice is a zero-shot text-to-speech (TTS) model designed for expressive long-form synthesis supporting 1–4 speakers in both monologue and dialogue settings, treating full conversations as a single generation problem rather than stitching isolated turns.
• SwanData-Speech is a data processing pipeline that converts ~2.59 million hours of in-the-wild audio into monologue and dialogue training subsets, featuring speaker-aware segmentation, pause-aware alignment via a novel Swan Forced Aligner, and pronunciation-hard synthetic data (RobustMegaTTS3).
• The model architecture combines a 25 Hz VAE, raw-text conditioning with pause symbols and pinyin substitution, and a flow-matching DiT with speaker-turn conditioning, trained via a three-stage curriculum (monologue → mixed → real dialogue) followed by DiffusionNFT post-training with phone-level and speaker-similarity rewards.
• On SwanBench-Speech, SwanVoice achieves the highest richness (3.81 monologue / 3.62 dialogue) and hierarchy (3.62 monologue / 3.71 dialogue) scores among all evaluated open-source baselines, though content accuracy remains its primary limitation.

---

Introduction and Theoretical Foundation

Background and Motivation

Recent zero-shot TTS systems have made single-speaker synthesis increasingly reliable, but applications such as short-form dramas and podcasts require treating multi-party conversations as one generation problem. The common workaround—synthesizing each turn separately and concatenating waveforms—produces results that sound "assembled rather than recorded as a scene," with inconsistencies in room response, background ambience, speaking intensity, and pause timing across turns.

Key Challenges

The paper identifies two central bottlenecks:

1. Dialogue data requirements: Expressive long-form synthesis needs speaker-consistent segments, pause-aware transcripts, quality filtering, and sufficient non-neutral speech. A speaker split error can corrupt turn control, while written-style punctuation can teach the model incorrect prosody.

2. Monologue preservation: Dialogue training should not degrade monologue ability. Many dialogue models fine-tune on dialogue data with speaker-switch labels, which often improves turn control but weakens monologue quality.

Architectural Context

Modern zero-shot TTS systems combine speech representations, neural vocoders, Transformer-based text/audio modeling, and generative modules (diffusion or flow matching). They divide into:

• Autoregressive (AR) formulations: e.g., CosyVoice, IndexTTS2 — suffer from sequential latency and exposure-bias failures (word skipping, repetition) in long dialogue
• Non-autoregressive (NAR) formulations: e.g., F5TTS, MegaTTS3 — better suited for long-form generation as they condition on the full text and speaker-turn sequence at once

---

Methodology

1. Data Processing Pipeline: SwanData-Speech

The pipeline processes ~2.59M hours of raw audio (2.24M hours Chinese, 0.35M hours English) through:

Segmentation and Speaker-Aware Processing:

• Speech enhancement: Vocal separation tool isolates the vocal component
• Speaker diarization: Uses 3D-Speaker toolkit with FSMN-Monophone VAD and CAM++ with spectral clustering; merges adjacent short segments (≤2s silence) from the same speaker; dialogue segments merged up to 120s with 2–4 speakers using sliding-window greedy merging

Transcription and Alignment:

• ASR: SenseVoice-Small for transcription, with ITN disabled; speaker turns wrapped with <S{id}> and </S{id}> tokens
• Punctuation correction: Pauses are detected via forced alignment timestamps:
  • < 0.08s: ignored
  • 0.08–0.18s: insert <|sp|>
  • 0.18–0.45s: use comma

  • 0.45s: use period/exclamation/question mark

Data Filtering: DNSMOS, non-intrusive PESQ/STOI (torchaudio-SQUIM), and emotion2vec+ for high-expressiveness subset selection.

RobustMegaTTS3: Synthetic pronunciation-hard data covering polyphonic characters, erhua, tone sandhi, homographs, stress shifts, and 100K Chinese–English code-switching texts, synthesized with MegaTTS 3.

2. Model Architecture: SwanVoice

VAE: Encoder $E$ maps waveform $s$ to latent $z$; decoder $D$ reconstructs as $\hat{s} = D(z) = D(E(s))$. Training objective:

$$\mathcal{L} = \mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{KL}} + \mathcal{L}_{\mathrm{Adv}}$$

where $\mathcal{L}_{\mathrm{rec}} = \|\Phi(s) - \Phi(\hat{s})\|_2^2$ is the spectrogram-domain reconstruction loss, $\mathcal{L}_{\mathrm{KL}}$ is a lightly-weighted KL regularizer, and $\mathcal{L}_{\mathrm{Adv}}$ is an LSGAN adversarial loss. Compression rate: 25 latent frames per second.

Tokenizer: CosyVoice BPE-based tokenizer with raw text input, augmented with:

• Dedicated pause token <|sp|>
• 1,549 pinyin syllable combinations for Chinese pronunciation control
• Speaker-turn label sequence aligned with text tokens

Flow-based Transformer: A DiT with RMSNorm and AdaLN-based global adapters. The flow-matching objective:

$$\mathcal{L}_{\mathrm{flow}} = \mathbb{E}_{t \sim \mathcal{U}(0,1), \mathbf{z}^{\star} \sim p_{\mathrm{data}}, \epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})} \left[ \|\mathbf{u}_{\theta}(\mathbf{z}_t, t, \mathbf{c}) - (\mathbf{z}^{\star} - \boldsymbol{\epsilon})\|_2^2 \right]$$

where $\mathbf{z}_t = (1-t)\boldsymbol{\epsilon} + t\mathbf{z}^{\star}$ and $\mathbf{c}$ includes text tokens, speaker-turn embeddings, and reference speech latent.

3. Curriculum Learning

4. Post-Training with DiffusionNFT

Reward models:

• Phone consistency reward: Based on WER computed on phone-tone pairs:

$$\mathrm{WER}(\mathbf{u}^{\mathrm{ref}}, \mathbf{u}^{\mathrm{hyp}}) = \frac{S + D + I}{\max(1, |\mathbf{u}^{\mathrm{ref}}|)}$$ $$r_{\mathrm{phone}} = \exp(-\mathrm{WER}(\mathbf{u}^{\mathrm{ref}}, \mathbf{u}^{\mathrm{hyp}}))$$

• Speaker similarity reward:

$$r_{\mathrm{sim}}(\hat{x}, x^{\mathrm{ref}}) = \cos(f_{\mathrm{spk}}(\hat{x}), f_{\mathrm{spk}}(x^{\mathrm{ref}}))$$

• Aggregated reward: $r = \frac{1}{2}(r_{\mathrm{phone}} + r_{\mathrm{sim}})$

Policy optimization (DiffusionNFT-style): Uses prompt-wise advantage $A_i = r_i - \bar{r}$ with soft preference weights $w_i$, positive/implicit-negative denoising branches, and reference-policy regularization:

$$\mathcal{L} = \mathcal{L}_{\mathrm{NFT}} + \lambda_{\mathrm{ref}}\mathcal{L}_{\mathrm{ref}}$$

5. Inference

• Staircase classifier-free guidance with three conditioning variants (null, text-only, full):

$$\tilde{v}_t = v_{\emptyset} + \omega_{\mathrm{text}}(v_{\mathrm{text}} - v_{\emptyset}) + \omega_{\mathrm{ref}}(v_{\mathrm{full}} - v_{\mathrm{text}})$$

• Sway sampling for coarse-to-fine generation
• Speaking-rate heuristic for duration estimation

---

Empirical Validation / Results

Evaluation Framework (SwanBench-Speech)

Three axes: Acoustics (timbre consistency via speaker embedding similarity, reverb consistency via SRMR standard deviation, sound fidelity via SQUIM-PESQ), Semantics (content error via CER/WER, prosodic coherence via SpeechJudge), and Expressiveness (richness via 10s-chunk MLLM scoring, hierarchy via Gemini-3-Pro on emotional variation, vocal dynamics, scene appropriateness).

Monologue Results (Table 1)

SwanVoice achieves the highest richness (3.81) and hierarchy (3.62), with gains of 0.39 and 0.56 points over VibeVoice (strongest baseline). Content error (0.172) is above the baseline average but not the best.

Dialogue Results (Table 2)

For dialogue, SwanVoice achieves richness/hierarchy of 3.62/3.71, outperforming the strongest baselines by 0.53/0.56 points.

Swan Forced Aligner Results (Table 3)

Swan Forced Aligner achieves the best open-source AAS (accumulated averaging shift) on Chinese and LibriSpeech-Clean benchmarks, within 0.18 ms of Qwen3 Forced Aligner on LibriSpeech-Others.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Full-context dialogue modeling: SwanVoice demonstrates that treating long-form dialogue as a single generation problem, rather than stitching isolated turns, is crucial for expressive coherence—the approach yields significantly higher richness and hierarchy scores than turn-by-turn approaches.

2. Pause-aware data construction: The paper shows that semantic punctuation correlates weakly with acoustic pauses, and that correcting punctuation to match acoustic evidence is essential for prosody control in TTS training—a finding with implications for all speech-data annotation pipelines.

3. Curriculum design: The three-stage training curriculum (monologue → mixed → real dialogue) addresses the fundamental tension between learning speech-text alignment and preserving monologue quality, providing a template for training multi-speaker models.

4. Value-free RL post-training: The DiffusionNFT-based approach demonstrates that online RL with phone-level and speaker-similarity rewards can improve robustness without requiring value models or full latent trajectory supervision.

Practical Implications

• Applications: Short-form dramas, podcasts, audiobooks, and multi-speaker content generation can benefit from a single model that handles both monologue and dialogue without stitching artifacts.
• Data pipeline: The SwanData-Speech pipeline offers a practical recipe for converting in-the-wild audio (podcasts, radio dramas, film/TV) into usable TTS training data, including the Swan Forced Aligner which provides state-of-the-art open-source alignment.
• Pronunciation robustness: The pinyin-hint mechanism and RobustMegaTTS3 synthetic data address Chinese polyphonic character disambiguation and code-switching challenges.

---

Conclusion

Main Takeaways

SwanVoice treats long-form dialogue as a full-context generation problem, achieving higher expressiveness (richness and hierarchy) than all evaluated open-source baselines in both monologue and dialogue settings. The data pipeline—with speaker-aware segmentation, pause-aware alignment, pronunciation hard cases, and emotion-based filtering—directly contributes to this behavior by addressing failure modes that become audible in long speech.

Limitations

• Content accuracy remains weaker than the best baselines in several settings
• Speaker switching can fail when speakers are acoustically close or when the prompt is short

Future Directions

1. Pronunciation control: Improving handling of difficult words and polyphonic characters
2. Alignment and pause modeling: More robust speech-text alignment and pause control
3. Speaker-turn conditioning: More robust conditioning for close voices and short prompts

The authors conclude that future work should focus on making long-form speech generation more reliable across these dimensions.