Summary (Overview)

• Dual contribution: This paper addresses two coupled gaps in streaming audio-visual target speaker extraction (AVTSE): the lack of realistic benchmarks and the computational heaviness of high-performance causal architectures.
• RealSSA benchmark: A new realistic AVTSE benchmark comprising RealSSA-Sim (controllable simulated scenes with 1–5 near-field and 10–15 far-field interfering speakers, reverberation, and sudden noise) and RealSSA-Real (a held-out real-recorded evaluation set).
• Falcon model: A novel causal AVTSE method featuring a single-pass multi-scale Separator that performs time-frequency separation in one encoder–decoder traversal, avoiding the multi-iteration recurrence of lightweight alternatives.
• Encoder-space speaker contrastive loss: A new training objective that suppresses near-field speaker leakage at zero inference cost and transfers across different AVTSE backbones.
• State-of-the-art results: Falcon achieves the best extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix, while reducing parameters by 91.9%, computation by 20.2×, and GPU inference latency by 11.1× relative to the causal AV-TFGridNet baseline.

Introduction and Theoretical Foundation

AVTSE leverages a speaker's visual stream (typically lip movements) as enrollment cues to separate the target speaker's voice from a multi-speaker mixture. The authors identify two fundamental gaps in current causal AVTSE systems:

1. Data gap: Existing benchmarks (LRS2-2Mix, LRS3-2Mix, VoxCeleb2-2Mix) fix the number of speakers at two, draw mixtures at controlled SNRs, and hold room acoustics and background noise constant. They fail to model:

   • Near-/far-field interference geometry
   • Reverberation effects
   • Sudden noise events
   • Variable speaker counts (1–5+)
   • Real-world deployment conditions (no real-recorded evaluation sets)

2. Model gap: High-performance causal architectures (AV-TFGridNet, AV-Mossformer2) are too heavy for edge deployment, while lightweight alternatives (SwiftNet) reclaim capacity by repeatedly invoking a small separator, which multiplies per-frame computation and erodes latency savings.

The theoretical foundation builds on:

• Signal model: The mixture is modeled as $\mathbf{x} = \mathbf{s} + \sum_{k=1}^{K} \mathbf{b}_k + \mathbf{f} + \mathbf{n}$, where $\mathbf{s}$ is target speech, $\mathbf{b}_k$ are near-field interferers, $\mathbf{f}$ is far-field interference, and $\mathbf{n}$ is environmental noise.
• Limitation of SI-SNR: Standard reconstruction losses (SI-SNR) penalize deviation from the clean target but don't explicitly push the estimate away from near-field speakers, leaving residual leakage that grows with the number of active near-field speakers.

Methodology

RealSSA Dataset

RealSSA-Sim provides four acoustic scene types with controllable parameters:

The generation pipeline: (1) Room simulation with gpuRIR, (2) Source placement (near-field ≤ 3m, far-field > 3m), (3) Mixing with EBU R128 loudness normalization (−23 LUFS) and peak limiting to −1 dBFS. RealSSA-Sim contains 100,826 training, 9,133 validation, and 9,169 test samples (~153 hours).

RealSSA-Real is a real-recorded evaluation set (~84 minutes, 10 participants, 8 environments) capturing natural acoustic conditions with a single mobile device.

Falcon Architecture

Overall pipeline: The system processes a mixture waveform $\mathbf{x} \in \mathbb{R}^{L}$ and mouth video $\mathbf{m} \in \mathbb{R}^{H \times W \times T_v}$ through:

1. Frozen FRCNN visual backbone → visual bottleneck → $\mathbf{V}_0 \in \mathbb{R}^{C_v \times T_v}$
2. Audio encoder $E_a$: STFT → 3-channel tensor (real, imaginary, magnitude) → 2D conv → $\mathbf{A}_0$
3. Audio/Video refinement blocks → fusion $\mathbf{F} \in \mathbb{R}^{C \times T \times N_f}$
4. Separator $B_a$: processes fused feature $\mathbf{F}$ → target speaker feature $\mathbf{E}$
5. Complex masking: The masked embedding is obtained via complex multiplication:

$$\hat{\mathbf{A}} = \begin{bmatrix} \mathbf{M}_r \odot \mathbf{A}_{0,r} - \mathbf{M}_i \odot \mathbf{A}_{0,i} \\ \mathbf{M}_r \odot \mathbf{A}_{0,i} + \mathbf{M}_i \odot \mathbf{A}_{0,r} \end{bmatrix} \in \mathbb{R}^{C_e \times T \times N_f}, \tag{1}$$

Single-Pass Multi-Scale Separator: An asymmetric U-Net where:

• Encoder has $L=4$ levels with depthwise convolutions (stride-1 3×3 at level 0, stride-2 5×5 at levels 1–3)
• Time-frequency modeling concentrated at the bottleneck via Time-Frequency Refinement Blocks (TFRBs)
• Decoder restores resolution coarse-to-fine with level-specific TFRBs at each scale

Each TFRB is a cascade of three stages:

$$\mathcal{R}_l(\mathbf{X}) = \mathcal{A}_l^{2D}\left(\mathcal{T}_l\left(\mathcal{F}_l^{rnn}(\mathbf{X})\right)\right), \tag{5}$$

where $\mathcal{F}_l^{rnn}$ is a bidirectional LSTM along frequency, $\mathcal{T}_l$ is a unidirectional LSTM along time, and $\mathcal{A}_l^{2D}$ is 2D self-attention with block-causal masking.

Global Context Injection (GCI) module injects bottleneck context into each skip:

$$\tilde{\mathbf{S}}_l = \mathcal{I}_l(\mathbf{S}_l, \mathbf{G}) = \phi_{loc}(\mathbf{S}_l) \odot \sigma(\phi_{gate}(\mathbf{G})) + \phi_{glb}(\mathbf{G}), \tag{6}$$

Streaming-causal layer normalization (LN_c) is defined per time-frequency position:

$$[\mathrm{LN}_c(\mathbf{Z})]_{c,t,f} = \gamma_c \cdot \frac{Z_{c,t,f} - \mu_{t,f}}{\sqrt{\sigma_{t,f}^2 + \epsilon}} + \beta_c, \quad \mu_{t,f} = \frac{1}{C_h} \sum_c Z_{c,t,f}, \tag{2}$$

Encoder-Space Speaker Contrastive Loss

The contrastive loss reuses the audio encoder $E_a$ for both query and references (with stop-gradient on references):

$$\mathbf{A}^+ = \operatorname{sg}(E_a(\mathbf{s})), \quad \mathbf{A}_k^- = \operatorname{sg}(E_a(\mathbf{b}_k)), \quad k = 1, \dots, K. \tag{8}$$

The loss combines positive attraction with softmax-weighted, margin-based repulsion:

$$\mathcal{L}_{spk} = \left(1 - \cos(\mathbf{q}, \mathbf{p})\right) + \sum_{k=1}^{K} \frac{\exp(\tau s_k)}{\sum_{j=1}^{K} \exp(\tau s_j)} \operatorname{ReLU}(s_k - m), \tag{10}$$

where m = 0.5 (60° angular margin) and τ is linearly annealed from 0 to 1.

The final objective is $\mathcal{L} = \mathcal{L}_{recon} + \lambda_{spk}\mathcal{L}_{spk}$ with $\mathcal{L}_{recon} = -\operatorname{SI-SNR}(\hat{\mathbf{s}}, \mathbf{s})$ and $\lambda_{spk} = 1.0$.

Empirical Validation / Results

Separation Quality (Table 3)

Speech Quality and Target-Speaker Consistency (Table 4)

Computational Efficiency (Table 5)

Generalization (Table 6)

Training on RealSSA-Sim (vs. VoxCeleb2) improved DNS-MOS for all five architectures and NISQA for four. Falcon improved from 2.431 → 3.615 NISQA and 2.878 → 3.276 DNSMOS.

Ablation Results

• Downsampling depth: 3 stages (vs. 1) improved SI-SNRi by 0.6 dB (7.1 → 7.7)
• Contrastive loss: Improved Falcon by 0.4 dB, SwiftNet-12 by 0.2 dB, Dolphin by 0.3 dB SI-SNRi
• τ annealing: Annealing 0→1 outperformed fixed schedules (7.7 vs. 7.4–7.5 SI-SNRi)

Theoretical and Practical Implications

1. Benchmark design: RealSSA is the first AVTSE benchmark satisfying all three axes jointly (speaker configuration, acoustic conditions, evaluation realism). This enables more reliable evaluation of causal models for real deployment.

2. Architecture efficiency: Falcon demonstrates that high-quality AVTSE doesn't require heavy architectures. Its single-pass multi-scale design achieves state-of-the-art quality with:

   • 91.9% fewer parameters than AV-TFGridNet
   • 20.2× less computation
   • 11.1× lower GPU inference latency This supports practical deployment on edge devices for latency-sensitive applications (video conferencing, hearing aids).

3. Training objective: The encoder-space contrastive loss provides an architecture-independent regularization signal that:

   • Suppresses near-field leakage explicitly
   • Adds zero inference cost (training-only)
   • Transfers across different backbones (validated on Falcon, SwiftNet-12, Dolphin)

4. Real-world generalization: Systematic acoustic coverage (variable speaker counts, near/far-field geometry, reverberation, sudden noise) transfers better to real recordings than large-scale but homogeneous corpora.

Conclusion

This paper makes two key contributions to streaming AVTSE:

1. RealSSA: A realistic benchmark combining controllable simulation (RealSSA-Sim) with held-out real recordings (RealSSA-Real), enabling evaluation under deployment-like conditions.

2. Falcon: A causal AVTSE method that achieves state-of-the-art extraction quality across all tested benchmarks while being dramatically more efficient than prior high-performance causal systems, supporting progress toward low-latency on-device AVTSE.

Limitations and future directions:

• Single-channel operation (multi-channel array simulation left for future extension)
• RealSSA-Real is relatively small (~84 minutes, 10 participants, 8 environments)
• The contrastive loss requires clean per-source waveforms, limiting adaptation to unlabeled real recordings
• Future work could explore multi-channel extensions, larger real-recorded datasets, and self-supervised adaptation of the contrastive loss