Summary (Overview)

• Dynamic Band Splitting (DBS) is proposed as an end-to-end framework that adapts frequency band boundaries to each input mixture while keeping the total number of band tokens fixed, addressing a key limitation of static band-split architectures in audio source separation.
• DBS achieves substantial gains across multiple backbones and datasets: DBS-BSRNN improves average SI-SDR by 29.0% on DnR v2 (7.69→9.07 dB), four-stem SDR by 6.8% on MUSDB18-HQ (5.09→5.43 dB), and SDRi by 3.6% on EchoSet.
• The method introduces content-conditioned boundary prediction using Hard Top-K selection within fixed frequency blocks, enabling mixture-dependent frequency partitioning under a fixed band-token budget.
• A straight-through estimator (STE) enables end-to-end training through the non-differentiable hard boundary selection, with a probability-guided smoothing mechanism for token expansion back to full STFT resolution.
• Ablation studies confirm that per-block budgets, learned spectral features, acoustic transition priors, and frequency refinement are all critical components, with each removal causing significant performance degradation.

Introduction and Theoretical Foundation

Audio source separation aims to recover individual sources (speech, music, environmental sounds) from mixtures, supporting applications like music remixing, hearing assistance, and meeting transcription. The fundamental challenge is preserving fine spectral structure while maintaining tractable computational cost.

The band-split paradigm was introduced by BSRNN (Luo & Yu, 2023), which groups STFT frequency bins into manually specified non-uniform bands, encoding each band as a fixed-dimensional token before modeling temporal and inter-band dependencies. Subsequent architectures (BS-RoFormer, Mel-Band RoFormer, TIGER) follow this pattern with predefined frequency layouts.

The core limitation: These methods apply the same frequency layout to every mixture. Since informative spectral structure varies with source content, fixed boundaries can:

• Merge discriminative spectral components into single representations
• Waste tokens on frequency regions with little informative content
• Fail to adapt to heterogeneous acoustic content across mixtures

The research question: How can frequency resolution be allocated adaptively to each mixture while keeping the number of band representations (and thus computational cost) fixed?

Theoretical foundation: The approach draws on:

• Hard Top-K selection for discrete boundary selection with fixed cardinality constraints
• Straight-through estimators (Bengio et al., 2013) for gradient propagation through non-differentiable operations
• Frequency-block scaffolding to guarantee each spectral region receives a prescribed share of the budget

Methodology

3.1 Dynamic Band Splitting

Given a complex spectrogram $\mathbf{X} \in \mathbb{C}^{F \times T}$ with $F$ frequency bins and $T$ time frames, DBS:

1. Divides the frequency axis into $R$ fixed, contiguous blocks $\{\mathcal{F}_r\}_{r=1}^{R}$ with $F_r = |\mathcal{F}_r|$ bins each, assigning a budget of $J_r$ bands per block, with total budget $J = \sum_{r=1}^{R} J_r$.

2. Predicts boundary scores for each internal candidate position $f$:

   • Extracts three magnitude statistics per bin: temporal mean, peak-to-mean ratio, and standard-deviation-to-mean ratio
   • A BiLSTM summarizes fused features into contextual vectors
   • The score combines a feature term $u_r^{\text{feat}}(f)$ and a magnitude term $u_r^{\text{mag}}(f)$:
   $$u_r(f) = u_r^{\text{feat}}(f) + 0.15 \cdot u_r^{\text{mag}}(f)$$

3. Computes boundary probabilities via a learned scale $\tau_r$ and position bias $\beta_r(f)$:

$$p_r(f) = \sigma\left(\tau_r\left(u_r(f) - 0.5\right) + \beta_r(f)\right) \tag{1}$$

4. Selects boundaries via Hard Top-K:

$$\mathcal{B}_r = \{f_r^{\min}\} \cup \text{TopK}\left(\{p_r(f)\}_{f \in \mathcal{F}_r \setminus \{f_r^{\min}\}}, J_r - 1\right) \tag{2}$$

5. Assigns bins to bands via cumulative counting:

$$s_r(f) = \sum_{i=f_r^{\min}}^{f} b_r(i), \quad s_r(f) \in \{1, \dots, J_r\} \tag{3}$$

3.2 Band Tokenization and Separation

• Bins in each band are pooled with learned content-dependent weights:

$$\mathbf{Z}_{r,j}(t) = \sum_{f \in \mathcal{S}_{r,j}} \alpha_{r,j}(f) \mathbf{U}_r(f, t) \in \mathbb{R}^{D} \tag{5}$$

• The resulting token sequence $\mathbf{Z} \in \mathbb{R}^{J \times T \times D}$ feeds standard band-split backbones (BSRNN, TIGER, BS-RoFormer, MelRoFormer).

3.3 Token Expansion & Mask Decoding

• Probability-guided smoothing: Interpolates separated tokens using averaged boundary probabilities $\bar{p}_{r,j}$ to soften transitions
• Token expansion: Copies smoothed tokens back to STFT bins using the hard assignment:

$$\mathbf{Z}_r^{e}(f, t) = \bar{\mathbf{Z}}_{r, s_r(f)}(t) \in \mathbb{R}^{D_{\text{sep}}} \tag{6}$$

• Frequency refinement: A residual frequency-direction block mixes neighboring bin features across band boundaries
• Frequency-shared mask prediction: Maps refined features to complex mask parameters, producing the estimated source spectrogram:

$$\hat{\mathbf{S}}_k = \mathbf{X} \odot \mathbf{M}_k \in \mathbb{C}^{F \times T} \tag{7}$$

3.4 End-to-End Training

• Straight-through estimator for gradient flow through hard boundary decisions:

$$\mathbf{Z}_r^{\text{STE}}(f) = \mathbf{Z}_r^{e}(f) + \mathbf{Z}_r^{\text{soft}}(f) - \text{sg}\left(\mathbf{Z}_r^{\text{soft}}(f)\right) \in \mathbb{R}^{T \times D_{\text{sep}}} \tag{8}$$

• Boundary density regularization:

$$\mathcal{L}_{\text{ratio}} = \frac{1}{R} \sum_{r=1}^{R} \left(\frac{1}{F_r} \sum_{f \in \mathcal{F}_r} p_r(f) - \frac{1}{\rho_r}\right)^{2} \tag{9}$$

• Joint objective:

$$\mathcal{L} = \mathcal{L}_{\text{sep}} + \lambda_{\text{ratio}} \mathcal{L}_{\text{ratio}} \tag{10}$$

Empirical Validation / Results

DnR v2 (Cinematic Audio Separation)

MUSDB18-HQ (Music Separation)

EchoSet (Speech Separation)

• DBS-BSRNN: +0.46 dB SDRi, +0.43 dB SI-SDRi over published BSRNN baseline
• DBS-TIGER: +0.33 dB SDRi, +0.31 dB SI-SDRi over published TIGER baseline

Isolating Dynamic Boundary Effects (DnR v2)

Design Ablations

Block structure: 4 blocks (9.07 dB) > 2 blocks (8.32 dB) > 1 block (6.30 dB), confirming per-block budgets prevent boundary concentration at low frequencies.

Boundary prediction components:

Boundary Adaptation Analysis

• All 64 test windows yielded distinct partitions
• Boundaries changed between adjacent windows within recordings
• Repeated inference on identical inputs produced identical boundaries (deterministic)

Theoretical and Practical Implications

Theoretical contributions:

1. Decoupling frequency resolution from output resolution: DBS separates the separator's internal frequency resolution from the STFT grid used for reconstruction, enabling adaptive allocation without sacrificing output fidelity.
2. Content-conditioned discrete optimization: The Hard Top-K + STE framework provides a principled way to learn discrete structural decisions (band boundaries) end-to-end within a fixed budget constraint.
3. General framework: DBS works across diverse backbones (BSRNN, TIGER, BS-RoFormer, MelRoFormer) and spectral bandwidths (0–22.05 kHz for music/cinematic, 0–8 kHz for speech), suggesting broad applicability.

Practical implications:

1. Improved separation quality: Consistent gains across all sources (music, speech, sound effects, vocals, drums, bass, other) without increasing token count.
2. Parameter efficiency: DBS variants often reduce parameters (e.g., DBS-BSRNN: 4.7M vs 35.5M static on DnR v2) while improving quality.
3. Computational trade-off: MACs increase somewhat (e.g., DBS-BSRNN: 124.0 vs 76.6 G/s on DnR v2) due to boundary prediction and frequency refinement, but the separator's core cost remains bounded by the fixed token budget.
4. Adaptive spectral allocation: The method naturally allocates more boundaries where spectral content is complex (e.g., low frequencies for music) and fewer where resolution is less critical.

Conclusion

DBS introduces a novel paradigm for band-split audio source separation: instead of fixing frequency boundaries as a structural prior, it learns to allocate a fixed band-token budget adaptively to each mixture's spectral content. The framework achieves consistent improvements across multiple backbones and datasets, demonstrating that content-adaptive frequency partitioning benefits source separation under computational constraints.

Key takeaways:

1. Mixture-dependent frequency partitioning improves separation quality over static layouts
2. Per-block budgets prevent pathological boundary concentration
3. Learned spectral features, acoustic transition priors, and frequency refinement are all essential components
4. The framework generalizes across music, speech, and cinematic audio separation

Future directions:

• Variable band budgets: The authors note that learnable-count variants tended to select maximal budgets, suggesting the need for explicit complexity constraints to enable adaptive token counts.
• Intra-window adaptation: Currently one partition is selected per inference window; adapting boundaries within windows could further improve performance.
• Broader applications: The framework could extend to other spectral processing tasks beyond source separation.

Limitations: DBS does not adapt representation size or boundary positions within an inference window, and variable-budget routing requires further research to avoid trivial solutions.