MACE-Dance: Motion-Appearance Cascaded Experts for Music-Driven Dance Video Generation

Summary (Overview)

• Framework: Introduces MACE-Dance, a cascaded Mixture-of-Experts (MoE) framework for music-driven dance video generation, decoupling motion and appearance synthesis.
• Motion Expert: Uses a Diffusion Model with a BiMamba-Transformer hybrid architecture and Guidance-Free Training (GFT) strategy to generate kinematically plausible and artistically expressive 3D dance motion from music.
• Appearance Expert: Adopts a decoupled Kinematic–Aesthetic fine-tuning strategy on Wan-Animate to synthesize high-fidelity videos from 3D motion and a reference image.
• Dataset & Protocol: Curates a large-scale dataset (MA-Data) and introduces a motion–appearance evaluation protocol to benchmark the task.
• Performance: Achieves state-of-the-art (SOTA) performance in music-driven dance video generation, as well as in the subtasks of music-driven 3D dance generation and pose-driven image animation.

Introduction and Theoretical Foundation

Music-driven dance video generation is a timely research direction with the rise of online dance platforms and advances in AIGC. The task faces two key challenges: (1) generating kinematically plausible and artistically expressive dance motions, and (2) achieving high-fidelity visual appearance with strong spatiotemporal consistency. Existing approaches from related domains (music-driven 3D dance generation, pose-driven image animation, audio-driven talking-head synthesis) are not readily transferable due to fundamental mismatches. Research on music-driven dance video generation itself remains limited and often fails to capture the inherently 3D nature of dance, compromising motion and appearance quality.

MACE-Dance addresses these challenges through a cascaded expert design. The Motion Expert generates 3D motion from music, enforcing kinematic plausibility and artistic expressiveness. The Appearance Expert synthesizes videos conditioned on this 3D motion and a reference image, preserving visual identity and spatiotemporal coherence. Crucially, the framework uses 3D SMPL parameters as the intermediate representation, rather than 2D keypoints, for three reasons:

1. Richer spatial fidelity: Preserves full-body geometric structure, including global translation and orientation.
2. Cleaner supervision: Disentangles pose from camera viewpoint and subject-specific appearance.
3. Better robustness: More robust to self-occlusion and viewpoint variation.

Methodology

The overall objective is to synthesize dance videos $D \in R^{T \times H \times W \times 3}$ given a music sequence $M \in R^{T \times C_m}$ and a reference image $I \in R^{H \times W \times 3}$.

Motion Expert

Generative Strategy: Uses a Diffusion Model with Guidance-Free Training (GFT).

• The forward noising process in DDPM is defined as:

$$q(z_t | x) \sim \mathcal{N}(\sqrt{\bar{\alpha}_t} x, (1 - \bar{\alpha}_t)I)$$

where $\bar{\alpha}_t \in (0,1)$ are constants following a monotonically decreasing schedule.

• With music conditioning $c$, the model learns to estimate $\hat{x}_\theta(z_t, t, c) \approx x$. GFT establishes $x_\beta$ as the new optimization target:

$$x_\beta = \beta \hat{x}_\theta(z_t, t, c, \beta) + (1 - \beta) \text{sg}[\hat{x}_\theta(z_t, t, \emptyset, 1)]$$

where $\emptyset$ denotes unconditional setting, $\text{sg}$ is stop-gradient, and $\beta \in [0,1]$ is a temperature parameter.

• The training loss combines reconstruction, 3D joint, velocity, and foot contact losses:

$$\mathcal{L} = \lambda_{\text{rec}}\mathcal{L}_{\text{rec}} + \lambda_{\text{joint}}\mathcal{L}_{\text{joint}} + \lambda_{\text{vel}}\mathcal{L}_{\text{vel}} + \lambda_{\text{foot}}\mathcal{L}_{\text{foot}}$$

where:

$$\mathcal{L}_{\text{rec}} = \mathbb{E}[\| x_\beta - x \|_2^2], \quad \mathcal{L}_{\text{joint}} = \mathbb{E}[\| FK(x_\beta) - FK(x) \|_2^2],$$ $$\mathcal{L}_{\text{vel}} = \mathbb{E}[\| FK(x_\beta)' - FK(x)' \|_2^2], \quad \mathcal{L}_{\text{foot}} = \mathbb{E}[\| FK(x_\beta)' \cdot \hat{b} \|_2^2]$$

$FK(\cdot)$ is the forward kinematic function, and $\hat{b}$ is the predicted binary foot contact label.

Model Architecture: Adopts a BiMamba–Transformer hybrid backbone.

• BiMamba captures intra-modal local dependencies in music or dance. The Selective State Space Model (Mamba) dynamics are:

$$h_t = \bar{A}_t h_{t-1} + \bar{B}_t x_t, \quad y_t = C_t h_t$$

where $\bar{A}_t, \bar{B}_t, C_t$ are dynamically updated. The state transitions are:

$$\bar{A} = \exp(\Delta A), \quad \bar{B} = (\Delta A)^{-1}(\exp(\Delta A) - I) \cdot \Delta B$$

• Transformer models cross-modal global context via cross-attention:

$$\text{Attention} = \text{softmax}\left(\frac{Q_d \cdot K_m^T}{\sqrt{C}}\right) V_m$$

where motion features are queries and music features provide keys/values.

• The architecture enables non-autoregressive generation of entire sequences, improving efficiency and avoiding exposure bias.

Appearance Expert

Built upon Wan-Animate, with a decoupled Kinematic–Aesthetic fine-tuning strategy to adapt it for dance video generation.

Model Architecture:

• A 3D-to-2D Motion Projector converts the SMPL sequence from the Motion Expert into 2D keypoints (using pyrender and ViTPose) for Wan-Animate.
• Kinematic Stage: Fine-tunes only the Body Adapter (freezing other components) to strengthen kinematic conditioning and motion adherence.
• Aesthetic Stage: Attaches LoRA adapters to each DiT block for aesthetic refinement, enhancing texture fidelity and stylistic consistency while preserving pretrained priors. LoRA updates the weight matrix $W_0$ as:

$$W = W_0 + \Delta W = W_0 + AB$$

where $A \in R^{m \times r}$ and $B \in R^{r \times n}$ are low-rank matrices ($r \ll m, n$).

Empirical Validation / Results

Dataset: MA-Data

A large-scale dance video dataset curated for benchmarking.

• Size: 70k clips of 5–10 seconds each (116 hours total), spanning over 20 dance genres.
• Composition:
  1. 3D-rendered data (motion-centric): 20k clips (28 hours) derived from FineDance, rendered from 3D professional dancer motions.
  2. In-the-wild internet data (appearance-centric): 50k clips (88 hours) collected from TikTok/YouTube, emphasizing visual appearance.
• Test set: 200 5-second clips from high-engagement videos across multiple genres.

Evaluation Protocol

A motion–appearance evaluation protocol.

• Motion dimension: Assess fidelity, diversity, and synchronization from a Human-Kinematics perspective using 2D keypoints extracted by ViTPose.
  • Metrics: FID$_{k}$, FID$_{g}$, DIV$_{k}$, DIV$_{g}$, Beat Alignment Score (BAS).
• Appearance dimension: Uses VBench dance-specific metrics.
  • Metrics: Imaging Quality (IQ), Aesthetic Quality (AQ), Subject Consistency (SC), Background Consistency (BC), Motion Smoothness (MS), Temporal Flickering (TF).

Quantitative Results

Table 1: Quantitative comparison on MA-Data in Music-Driven Dance Video Generation

MACE-Dance achieves SOTA performance on both motion and appearance metrics.

Table 2: Quantitative comparison on FineDance in Music-Driven 3D Dance Generation

The Motion Expert achieves SOTA performance, with high fidelity, diversity, synchronization, and efficiency (770 FPS).

Table 3: Quantitative comparison on MA-Data in Pose-Driven Image Animation

The Appearance Expert achieves SOTA performance, validated by the two-stage fine-tuning ablation.

Qualitative Analysis

• Effect Comparison: MACE-Dance generates videos with kinematically plausible and artistically expressive motion, and spatiotemporally coherent appearance, outperforming baselines like Hallo2, EDGE, Lodge, MEGA, WAN-S2V, and Echomimic-V3 (see Fig. 3).
• Cross-Genre Generation: Effectively generates distinct genre-specific motions (Uyghur, Dunhuang, Dai, K-Pop, Popping) as shown in Fig. 4.
• Long-Sequence Generation: Produces coherent long-sequence dance videos (up to 30 seconds) thanks to the BiMamba–Transformer hybrid and pose-driven relay rendering (see Fig. 6).

Ablation Studies

Motion Expert Ablation (Table 2):

• BiMamba → Mamba: Removes bidirectional context, degrading dance quality metrics (FID$_k$=65.10, FID$_g$=51.74) though efficiency improves (1044 FPS). Generated dances become simpler.
• BiMamba → Transformer: Deprives non-autoregressive generation, causing collapse to in-place jitter. Metrics drop severely (FID$_k$=104.93, FID$_g$=114.42).
• GFT → CFG: Replacing GFT with classifier-free guidance leads to modest decline in metrics and lower generation efficiency.

Appearance Expert Ablation (Table 3 & Fig. 7):

• w/o. Kinematic Stage: Leads to modest metric decline and noticeable kinematic errors/motion blur.
• w/o. Aesthetic Stage: Causes substantial degradation and obvious ghosting artifacts.
• The full Appearance Expert outperforms Wan-Animate baseline.

Motion Representation (2D vs. 3D) (Table 4):

3D representation consistently outperforms 2D across both motion and final video metrics.

Role of Each Expert (Cross-Composition Analysis, Table 5):

Both experts contribute positively; the Motion Expert strengthens music-motion alignment (BAS), and the Appearance Expert improves visual quality (AQ, SC).

Comparison with Video Foundation Models (Table 6):

MACE-Dance achieves better overall performance, particularly in SC, FID, and BAS, indicating stronger music-motion alignment and visual quality.

Theoretical and Practical Implications

• Task Decoupling: The cascaded expert design effectively isolates motion semantics from visual appearance, reducing the complexity of learning a direct music-to-video mapping. The explicit 3D motion representation suppresses spurious cross-modal correlations and provides an interpretable intermediate interface.
• Architectural Innovations: The BiMamba–Transformer hybrid backbone combines local dependency modeling (BiMamba) with global cross-modal context (Transformer), enabling high-quality, non-autoregressive sequence generation.