# Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

> Falcon achieves state-of-the-art audio-visual target speaker extraction with 91.9% fewer parameters and 20.2x less computation than prior causal systems, validated on the new realistic RealSSA benchmark.

- **Source:** [arXiv](https://arxiv.org/abs/2610.32749)
- **Published:** 2026-10-30
- **Permalink:** https://picx.dev/p/r8N2ez
- **Whiteboard:** https://picx.dev/p/r8N2ez/image

## Summary

## Summary (Overview)

- **Dual contribution**: This paper addresses two coupled gaps in streaming audio-visual target speaker extraction (AVTSE): the lack of realistic benchmarks and the computational heaviness of high-performance causal architectures.
- **RealSSA benchmark**: A new realistic AVTSE benchmark comprising RealSSA-Sim (controllable simulated scenes with 1–5 near-field and 10–15 far-field interfering speakers, reverberation, and sudden noise) and RealSSA-Real (a held-out real-recorded evaluation set).
- **Falcon model**: A novel causal AVTSE method featuring a single-pass multi-scale Separator that performs time-frequency separation in one encoder–decoder traversal, avoiding the multi-iteration recurrence of lightweight alternatives.
- **Encoder-space speaker contrastive loss**: A new training objective that suppresses near-field speaker leakage at zero inference cost and transfers across different AVTSE backbones.
- **State-of-the-art results**: Falcon achieves the best extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix, while reducing parameters by 91.9%, computation by 20.2×, and GPU inference latency by 11.1× relative to the causal AV-TFGridNet baseline.

## Introduction and Theoretical Foundation

AVTSE leverages a speaker's visual stream (typically lip movements) as enrollment cues to separate the target speaker's voice from a multi-speaker mixture. The authors identify two fundamental gaps in current causal AVTSE systems:

1. **Data gap**: Existing benchmarks (LRS2-2Mix, LRS3-2Mix, VoxCeleb2-2Mix) fix the number of speakers at two, draw mixtures at controlled SNRs, and hold room acoustics and background noise constant. They fail to model:
   - Near-/far-field interference geometry
   - Reverberation effects
   - Sudden noise events
   - Variable speaker counts (1–5+)
   - Real-world deployment conditions (no real-recorded evaluation sets)

2. **Model gap**: High-performance causal architectures (AV-TFGridNet, AV-Mossformer2) are too heavy for edge deployment, while lightweight alternatives (SwiftNet) reclaim capacity by repeatedly invoking a small separator, which multiplies per-frame computation and erodes latency savings.

The theoretical foundation builds on:
- **Signal model**: The mixture is modeled as $\mathbf{x} = \mathbf{s} + \sum_{k=1}^{K} \mathbf{b}_k + \mathbf{f} + \mathbf{n}$, where $\mathbf{s}$ is target speech, $\mathbf{b}_k$ are near-field interferers, $\mathbf{f}$ is far-field interference, and $\mathbf{n}$ is environmental noise.
- **Limitation of SI-SNR**: Standard reconstruction losses (SI-SNR) penalize deviation from the clean target but don't explicitly push the estimate away from near-field speakers, leaving residual leakage that grows with the number of active near-field speakers.

## Methodology

### RealSSA Dataset

**RealSSA-Sim** provides four acoustic scene types with controllable parameters:

| Scene | Space size | RT60 | Near speakers | Far speakers | Env. noise SIR | Sudden noise |
|---|---|---|---|---|---|---|
| Small meeting | 4–6 × 3–5 × 2.5–3 m | 0.3–0.6 s | 1–5, 0.5–1.5 m | none | 5–15 dB | 30%, 1 event |
| Medium indoor | 8–15 × 8–12 × 3–4 m | 0.5–0.9 s | 1–3, 1.0–2.5 m | 10–15, 3–15 m | 5–10 dB | 40%, 1–2 events |
| Large public | 20–40 × 15–30 × 4–8 m | 1.0–2.0 s | 1–4, 1.0–3.0 m | 10–15, 5–20 m | 2–8 dB | 50%, 1–3 events |
| Outdoor | 50 × 50 × 20 m | 0.05–0.1 s | 1–4, 1.0–4.0 m | 10–15, 5–25 m | −2–5 dB | 40%, 1–3 events |

The generation pipeline: (1) Room simulation with gpuRIR, (2) Source placement (near-field ≤ 3m, far-field > 3m), (3) Mixing with EBU R128 loudness normalization (−23 LUFS) and peak limiting to −1 dBFS. RealSSA-Sim contains 100,826 training, 9,133 validation, and 9,169 test samples (~153 hours).

**RealSSA-Real** is a real-recorded evaluation set (~84 minutes, 10 participants, 8 environments) capturing natural acoustic conditions with a single mobile device.

### Falcon Architecture

**Overall pipeline**: The system processes a mixture waveform $\mathbf{x} \in \mathbb{R}^{L}$ and mouth video $\mathbf{m} \in \mathbb{R}^{H \times W \times T_v}$ through:

1. **Frozen FRCNN visual backbone** → visual bottleneck → $\mathbf{V}_0 \in \mathbb{R}^{C_v \times T_v}$
2. **Audio encoder** $E_a$: STFT → 3-channel tensor (real, imaginary, magnitude) → 2D conv → $\mathbf{A}_0$
3. **Audio/Video refinement blocks** → fusion $\mathbf{F} \in \mathbb{R}^{C \times T \times N_f}$
4. **Separator** $B_a$: processes fused feature $\mathbf{F}$ → target speaker feature $\mathbf{E}$
5. **Complex masking**: The masked embedding is obtained via complex multiplication:
$$\hat{\mathbf{A}} = \begin{bmatrix} \mathbf{M}_r \odot \mathbf{A}_{0,r} - \mathbf{M}_i \odot \mathbf{A}_{0,i} \\ \mathbf{M}_r \odot \mathbf{A}_{0,i} + \mathbf{M}_i \odot \mathbf{A}_{0,r} \end{bmatrix} \in \mathbb{R}^{C_e \times T \times N_f}, \tag{1}$$

**Single-Pass Multi-Scale Separator**: An asymmetric U-Net where:
- Encoder has $L=4$ levels with depthwise convolutions (stride-1 3×3 at level 0, stride-2 5×5 at levels 1–3)
- Time-frequency modeling concentrated at the bottleneck via Time-Frequency Refinement Blocks (TFRBs)
- Decoder restores resolution coarse-to-fine with level-specific TFRBs at each scale

Each TFRB is a cascade of three stages:
$$\mathcal{R}_l(\mathbf{X}) = \mathcal{A}_l^{2D}\left(\mathcal{T}_l\left(\mathcal{F}_l^{rnn}(\mathbf{X})\right)\right), \tag{5}$$
where $\mathcal{F}_l^{rnn}$ is a bidirectional LSTM along frequency, $\mathcal{T}_l$ is a unidirectional LSTM along time, and $\mathcal{A}_l^{2D}$ is 2D self-attention with block-causal masking.

**Global Context Injection (GCI)** module injects bottleneck context into each skip:
$$\tilde{\mathbf{S}}_l = \mathcal{I}_l(\mathbf{S}_l, \mathbf{G}) = \phi_{loc}(\mathbf{S}_l) \odot \sigma(\phi_{gate}(\mathbf{G})) + \phi_{glb}(\mathbf{G}), \tag{6}$$

**Streaming-causal layer normalization** (LN_c) is defined per time-frequency position:
$$[\mathrm{LN}_c(\mathbf{Z})]_{c,t,f} = \gamma_c \cdot \frac{Z_{c,t,f} - \mu_{t,f}}{\sqrt{\sigma_{t,f}^2 + \epsilon}} + \beta_c, \quad \mu_{t,f} = \frac{1}{C_h} \sum_c Z_{c,t,f}, \tag{2}$$

### Encoder-Space Speaker Contrastive Loss

The contrastive loss reuses the audio encoder $E_a$ for both query and references (with stop-gradient on references):
$$\mathbf{A}^+ = \operatorname{sg}(E_a(\mathbf{s})), \quad \mathbf{A}_k^- = \operatorname{sg}(E_a(\mathbf{b}_k)), \quad k = 1, \dots, K. \tag{8}$$

The loss combines positive attraction with softmax-weighted, margin-based repulsion:
$$\mathcal{L}_{spk} = \left(1 - \cos(\mathbf{q}, \mathbf{p})\right) + \sum_{k=1}^{K} \frac{\exp(\tau s_k)}{\sum_{j=1}^{K} \exp(\tau s_j)} \operatorname{ReLU}(s_k - m), \tag{10}$$
where m = 0.5 (60° angular margin) and τ is linearly annealed from 0 to 1.

The final objective is $\mathcal{L} = \mathcal{L}_{recon} + \lambda_{spk}\mathcal{L}_{spk}$ with $\mathcal{L}_{recon} = -\operatorname{SI-SNR}(\hat{\mathbf{s}}, \mathbf{s})$ and $\lambda_{spk} = 1.0$.

## Empirical Validation / Results

### Separation Quality (Table 3)

| Method | RealSSA-Real DNSMOS ↑ | RealSSA-Sim SI-SNRi ↑ | LRS2-2Mix SI-SNRi ↑ | LRS3-2Mix SI-SNRi ↑ | VoxCeleb2-2Mix SI-SNRi ↑ |
|---|---|---|---|---|---|
| AV-ConvTasNet | 2.580 | 1.2 | 9.8 | 12.5 | 6.9 |
| AV-Sepformer | 2.813 | 5.5 | 10.6 | 13.4 | 6.9 |
| AV-TF-GridNet | 3.030 | 7.6 | 12.7 | 14.3 | 5.8 |
| RTFSNet-12 | 3.066 | 4.9 | 12.4 | 14.0 | 8.4 |
| CTCNet | 2.840 | 5.4 | 8.0 | 9.4 | 2.5 |
| AV-DPRNN | 2.901 | 5.4 | 10.2 | 11.9 | 7.8 |
| AV-Mossformer2 | 2.912 | 6.0 | 13.2 | 16.2 | 7.7 |
| Swift-Net-12 | 3.276 | 7.3 | 13.9 | 16.0 | 9.4 |
| Dolphin | 2.960 | 6.7 | 14.0 | 14.3 | 9.2 |
| **Falcon (Ours)** | **3.276** | **7.7** | **14.1** | **16.4** | **10.5** |

### Speech Quality and Target-Speaker Consistency (Table 4)

| Method | RealSSA-Sim WER (%) ↓ | RealSSA-Sim Similarity ↑ | RealSSA-Real WER (%) ↓ | RealSSA-Real Similarity ↑ |
|---|---|---|---|---|
| AV-Sepformer | 32.64 | 38.69 | 34.61 | 38.25 |
| AV-TF-GridNet | 27.31 | 40.44 | 34.95 | 39.88 |
| RTFSNet-12 | 31.91 | 36.51 | 35.56 | 35.78 |
| CTCNet | 34.44 | 37.08 | 38.81 | 35.33 |
| AV-Mossformer2 | 32.44 | 37.34 | 32.00 | 36.42 |
| Swift-Net-12 | 28.59 | 41.79 | 33.72 | 40.04 |
| Dolphin | 49.48 | 24.98 | 39.06 | 35.95 |
| Ground truth | 11.83 | 64.69 | – | – |
| **Falcon (Ours)** | **25.47** | **43.05** | **32.74** | **42.23** |

### Computational Efficiency (Table 5)

| Method | Params (M) ↓ | MACs (G) ↓ | CPU Latency (ms) ↓ | GPU Latency (ms) ↓ | RAM (MB) ↓ | GPU Inf. (MB) ↓ |
|---|---|---|---|---|---|---|
| AV-TF-GridNet | 6.2 | 213.8 | 5646.3 | 567.8 | 136.3 | 1015.8 |
| AV-Mossformer2 | 56.7 | 224.2 | 16670.7 | 114.9 | 330.9 | 539.8 |
| Swift-Net-12 | 0.5 | 45.3 | 4163.8 | 172.1 | 482.3 | 411.7 |
| Dolphin | 5.6 | 18.1 | 2411.0 | 87.9 | 560.2 | 390.3 |
| **Falcon (Ours)** | **0.5** | **10.6** | **1361.2** | **51.3** | **125.3** | **364.7** |

### Generalization (Table 6)

Training on RealSSA-Sim (vs. VoxCeleb2) improved DNS-MOS for all five architectures and NISQA for four. Falcon improved from 2.431 → 3.615 NISQA and 2.878 → 3.276 DNSMOS.

### Ablation Results

- **Downsampling depth**: 3 stages (vs. 1) improved SI-SNRi by 0.6 dB (7.1 → 7.7)
- **Contrastive loss**: Improved Falcon by 0.4 dB, SwiftNet-12 by 0.2 dB, Dolphin by 0.3 dB SI-SNRi
- **τ annealing**: Annealing 0→1 outperformed fixed schedules (7.7 vs. 7.4–7.5 SI-SNRi)

## Theoretical and Practical Implications

1. **Benchmark design**: RealSSA is the first AVTSE benchmark satisfying all three axes jointly (speaker configuration, acoustic conditions, evaluation realism). This enables more reliable evaluation of causal models for real deployment.

2. **Architecture efficiency**: Falcon demonstrates that high-quality AVTSE doesn't require heavy architectures. Its single-pass multi-scale design achieves state-of-the-art quality with:
   - 91.9% fewer parameters than AV-TFGridNet
   - 20.2× less computation
   - 11.1× lower GPU inference latency
   This supports practical deployment on edge devices for latency-sensitive applications (video conferencing, hearing aids).

3. **Training objective**: The encoder-space contrastive loss provides an architecture-independent regularization signal that:
   - Suppresses near-field leakage explicitly
   - Adds zero inference cost (training-only)
   - Transfers across different backbones (validated on Falcon, SwiftNet-12, Dolphin)

4. **Real-world generalization**: Systematic acoustic coverage (variable speaker counts, near/far-field geometry, reverberation, sudden noise) transfers better to real recordings than large-scale but homogeneous corpora.

## Conclusion

This paper makes two key contributions to streaming AVTSE:

1. **RealSSA**: A realistic benchmark combining controllable simulation (RealSSA-Sim) with held-out real recordings (RealSSA-Real), enabling evaluation under deployment-like conditions.

2. **Falcon**: A causal AVTSE method that achieves state-of-the-art extraction quality across all tested benchmarks while being dramatically more efficient than prior high-performance causal systems, supporting progress toward low-latency on-device AVTSE.

**Limitations and future directions**:
- Single-channel operation (multi-channel array simulation left for future extension)
- RealSSA-Real is relatively small (~84 minutes, 10 participants, 8 environments)
- The contrastive loss requires clean per-source waveforms, limiting adaptation to unlabeled real recordings
- Future work could explore multi-channel extensions, larger real-recorded datasets, and self-supervised adaptation of the contrastive loss

---

_Markdown view of https://picx.dev/p/r8N2ez, served by PicX — AI-generated visual whiteboard summaries of research papers._