CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generation

Summary (Overview)

• CoInteract is an end-to-end framework for speech-driven human-object interaction (HOI) video synthesis that conditions on a person reference image, product reference image, text prompts, and speech audio, generating physically plausible and structurally stable interaction videos.

• The paper introduces Human-Aware Mixture-of-Experts (MoE), which routes tokens to lightweight region-specialized experts (head, hand, base) via spatially supervised routing, significantly improving fine-grained structural fidelity (especially hands and faces) with minimal parameter overhead.

• A novel Spatially-Structured Co-Generation paradigm jointly trains an RGB appearance stream and an auxiliary HOI structure stream within a shared Diffusion Transformer (DiT) backbone, using a texture-stripped silhouette-style structure stream to inject interaction geometry priors.

• A Two-Stage Asymmetric Co-Attention mechanism enables the HOI branch to be removed at inference, achieving zero additional computational cost while still transferring interaction-structure supervision to the RGB generator during training.

• CoInteract outperforms existing methods (AnchorCrafter, Phantom, Humo, VACE, InteractAvatar, SkyReels-V3) across multiple metrics, achieving best results in interaction plausibility (VLM-QA: 0.72), hand quality (HQ: 0.724), identity preservation (DINOid: 0.671, FaceSim: 0.696), and temporal coherence (Smooth: 0.9951).

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses the emerging need for Human-Object Interaction (HOI) video synthesis—a step beyond traditional talking-avatar generation. While speech-driven avatar generation has achieved photorealism, real-world applications such as e-commerce, digital advertising, and virtual marketing require active product demonstration with:

• Coordinated hand movements
• Precise object manipulation
• Strict physical plausibility (no hand-object interpenetration)
• Structural stability in sensitive regions (hands, faces)

Limitations of Existing Approaches

The authors identify two dominant paradigms in prior work:

1. Multi-condition generation: Methods like AnchorCrafter [51] and ByteLoom [26] extract per-frame human poses and object conditions to guide generation, but suffer from heavy preprocessing requirements and domain-specific signals that limit robustness and generalization.

2. Multi-reference generation: Methods like Humo [5], SkyReels-V3 [21], and VACE [17] condition on person/product references directly but lack explicit mechanisms to enforce interaction structure.

Core Theoretical Insight

"A model must not only 'see' pixels but also 'understand' the underlying structural and interaction relationships."

The authors argue that RGB-centric diffusion backbones have no built-in notion of 3D hand-object spatial relationships or body structure. Models trained purely on pixel-level supervision must rely on appearance cues alone, leading to two recurring failure modes:

1. Structural collapse in hands and faces (fingers merging, facial features blurring)
2. Physical violations such as human-object interpenetration (hands passing through product surfaces)

Technical Foundation

The framework builds on:

• Diffusion Transformer (DiT) backbones [33] for spatiotemporal token modeling
• Flow matching objectives for stable training
• 3D Rotary Positional Encoding (RoPE) for multi-modal coordinate assignment
• Mixture-of-Experts (MoE) architectures [8, 37] for region-specialized processing

Methodology

3.1 Unified RGB–HOI Co-Generation

The core architecture jointly trains two streams within a shared DiT backbone:

• RGB appearance stream $\mathbf{z}_r$: The standard video generation pathway
• HOI structure stream $\mathbf{z}_h$: A texture-stripped, silhouette-like 3-channel rendering (projected human mesh fused with object masks)

Joint Flow-Matching Objective

The training optimizes a joint flow-matching loss supervising both streams:

$$\mathcal{L}_{flow} = \mathcal{L}_{r} + \lambda_{h} \mathcal{L}_{h}$$ $$\begin{aligned} \mathcal{L}_{r} &= \mathbb{E}_{t, \mathbf{z}_{0}, \mathbf{z}_{1}} \Big[ \| \mathbf{v}_{r} - \mathbf{v}_{\theta}(\mathbf{z}_{r,t}, t, \mathbf{c}) \|_{2}^{2} \Big], \\ \mathcal{L}_{h} &= \mathbb{E}_{t, \mathbf{z}_{0}, \mathbf{z}_{1}} \Big[ \| \mathbf{v}_{h} - \mathbf{v}_{\theta}(\mathbf{z}_{h,t}, t, \mathbf{c}) \|_{2}^{2} \Big]. \end{aligned}$$

where $\mathbf{v}$ denotes the target velocity field, $t$ is the diffusion timestep, and $\mathbf{c}$ denotes conditioning (text, audio, dual reference images, and motion latents). The weight $\lambda_h = 1$ is used by default.

Multi-Modal Coordinate Assignment via 3D RoPE

Each token is assigned a 3D coordinate $(h, w, t)$ encoded by 3D Rotary Positional Encoding. Key design choices:

• Spatial coordinates: RGB stream uses $w \in [0, W]$ while HOI stream uses $w \in [-W, 0]$, sharing identical height and time indices to preserve pixel-level correspondence
• Temporal causality: Historical motion frames get negative temporal indices ($t \in \{-N, \dots, -1\}$)
• Reference anchoring: Static reference images are mapped to far-field temporal locations ($t = 30, 31$) as global identity anchors

Formally:

$$\mathrm{Pos}(x_{i,j,k}) = \mathrm{RoPE}_{3D}(h_i, w_j', t_k)$$

where $w_j'$ accounts for the virtual width shift in the HOI stream.

Two-Stage Asymmetric Co-Attention

The attention mask $\mathbf{M}$ is defined as:

$$\mathbf{M}_{i,j} = \begin{cases} 1, & \text{if } i \in \mathcal{T}_{r}, j \in \mathcal{T}_{r}, \\ 1, & \text{if } i \in \mathcal{T}_{h}, j \in \mathcal{T}_{r} \cup \mathcal{T}_{h}, \\ 0, & \text{otherwise}. \end{cases}$$

• Stage 1 (5K iterations): Full bidirectional attention across both streams for rapid convergence
• Stage 2 (2K iterations): Asymmetric mask where RGB queries attend only to RGB tokens (independent of HOI), while HOI queries attend to both streams

Since $\mathcal{L}_h$ backpropagates through the HOI←RGB cross-attention into shared DiT parameters, interaction-structure supervision transfers to the RGB generator even when the HOI branch is removed at inference.

3.2 Human-Aware Mixture-of-Experts

Four experts are used:

• Shared expert: Reuses the original DiT FFN as a shortcut path
• Three lightweight experts (Head, Hand, Base): Small FFNs with hidden dimension 256

Spatially Supervised Routing

A stop-gradient operation $sg[\cdot]$ prevents router optimization from interfering with DiT representation learning:

$$G(x_i) = \mathrm{Softmax}(\mathcal{W}_{g} \cdot sg[\mathbf{h}_i])$$

Face and hand bounding boxes supervise the router via cross-entropy:

$$\mathcal{L}_{route} = - \sum_{i} \sum_{k \in \{\text{head,hand,base}\}} \mathbb{1}(y_i = k) \log(G(x_i)_k)$$

Total Training Objective

$$\mathcal{L}_{total} = \mathcal{L}_{flow} + \eta \mathcal{L}_{route}$$

with $\eta = 1$.

3.3 Data Curation

• Raw HOI videos are transformed into paired RGB and HOI-structure representations
• Qwen-Edit [48] decouples entities into independent person and product references
• SAM3 [3] provides object masks; SAM3D-body [55] recovers human meshes (projected to image plane)
• Off-the-shelf detectors [29, 36] provide face and hand bounding boxes for MoE routing supervision

Empirical Validation / Results

Training Details

• Dataset: 40 hours of product demonstration and live-streaming videos, filtered to 12K high-quality clips; held-out test set of 50 clips
• Initialization: WanS2V [10] backbone
• Optimizer: AdamW with learning rate $1 \times 10^{-4}$ and cosine annealing
• Inference: CFG=5, 40 steps, 480p resolution

Quantitative Comparison

Table 1: Quantitative comparison on the HOI video generation test set. Bold = best, underline = second best, "—" = unsupported.

Key metrics:

• Video Quality: AES (LAION aesthetic predictor), IQ (MUSIQ), Smooth (CLIP cosine similarity between consecutive frames)
• HOI: VLM-QA (Gemini-3-Pro structured questionnaire), HQ (DWPose hand keypoint confidence)
• Reference Consistency: DINOid/DINOobj (DINOv2 cosine similarity), FaceSim (ArcFace)
• Audio-Visual: Syncconf (lip-sync confidence from SyncNet)

CoInteract achieves best results in VLM-QA (0.72) and HQ (0.724)—confirming superior interaction plausibility and hand structural stability. Phantom and Humo score slightly higher on AES because they hallucinate novel backgrounds that appear aesthetically pleasing but deviate from the input reference.

User Study

Table 2: User study (mean rank, lower is better).

24 evaluators, 10 randomly sampled test cases per evaluator, blind randomized order. CoInteract achieves the best (lowest) mean rank across all criteria, with the largest advantage on Interaction Plausibility.

Ablation Study

Table 3: Ablation study on core components.

Key findings:

• Removing MoE: HQ drops 0.724→0.658, FaceSim drops 0.696→0.662 (confirms role in fine-grained structural fidelity)
• Removing Co-Gen: VLM-QA drops most dramatically (0.72→0.48, −33.3%), showing the auxiliary stream is essential for physical interaction constraints
• Keeping HOI at inference (w/o Asym. Mask): Slightly better VLM-QA (0.76) and HQ (0.738) but 4.13× inference cost due to doubled token count
• MoE introduces only 1.04× overhead relative to MoE-free baseline

Theoretical and Practical Implications

Theoretical Contributions

1. Structure-aware generation paradigm: The paper challenges the assumption that pixel-level supervision suffices for physically plausible video generation. By demonstrating that a texture-stripped structural stream can regularize shared backbone weights, it provides evidence that hybrid supervision combining appearance and geometry produces more physically consistent outputs.

2. Asymmetric co-attention as a training-inference decoupling mechanism: The two-stage asymmetric mask design shows that structural priors can be injected during training and discarded at inference, addressing the common efficiency bottleneck of multi-stream architectures.

3. Spatially supervised routing in MoE: Using bounding box supervision for token routing demonstrates that explicit spatial priors can effectively guide expert specialization without destabilizing the shared representation learning.

Practical Implications

1. Zero-overhead deployment: The HOI branch is removed at inference, meaning CoInteract offers physical-plausibility improvements at no additional computational cost—critical for real-world deployment in e-commerce and advertising.

2. Unified end-to-end framework: Unlike methods requiring heavy inference-time preprocessing (e.g., per-frame pose extraction in AnchorCrafter), CoInteract accepts only reference images, text, and audio—simplifying the user pipeline.

3. Generalization to unseen objects: CoInteract demonstrates robustness on unseen product categories without requiring task-specific training data, addressing a key limitation of object-specific methods.

4. Applications: The framework is directly applicable to e-commerce product demonstrations, digital advertising, virtual marketing, and remote education.

Conclusion

Main Takeaways

• CoInteract introduces a novel structure-aware framework for HOI video synthesis that embeds human structural priors and interaction geometry constraints directly into the DiT backbone.

• The Human-Aware MoE with spatially supervised routing significantly improves hand and face fidelity with minimal parameter overhead (1.04× inference cost).

• The Spatially-Structured Co-Generation paradigm with asymmetric co-attention effectively reduces hand-object interpenetration and geometric misalignment while maintaining a zero-overhead inference path.

• Extensive experiments demonstrate consistent improvements over existing methods in interaction plausibility, structural stability, and identity preservation.

Future Directions

The paper's approach opens several avenues for future research:

1. Scaling to longer sequences: Extending the framework to handle longer, more complex interaction sequences
2. More detailed 3D priors: Incorporating more sophisticated 3D hand-object models beyond silhouette-based structure streams
3. Generalization to multi-object scenarios: Extending beyond single-person-single-object interactions
4. Real-time applications: Further optimization for real-time interactive use cases in virtual marketing and digital advertising