Summary (Overview)

• DONIC is the first generalist bimanual reference tracker that jointly trains across operation categories and human-motion sources, achieving state-of-the-art performance on unseen trajectories and objects without fine-tuning.
• BiHOI is a new multi-view hand–object interaction dataset with 1,575 sequences spanning 229 objects and 14.77 hours, emphasizing sustained bimanual contact (65.91% both-hand contact rate, exceeding prior datasets by 27+ percentage points).
• The method combines proximal policy optimization (PPO) with behavioral cloning (BC) in an iterative feedback loop, using an object-centered per-hand reward to handle asymmetric bimanual coordination.
• A novel Clip-Centered Advantages (CCA) mechanism removes reference-dependent advantage offsets while preserving within-trajectory differences, enabling stable joint training across heterogeneous references.
• A single checkpoint achieves 86.79% success on unseen trajectories and 57.32% on held-out objects, surpassing the strongest bimanual baseline by 77.36 and 45.73 percentage points, respectively.

---

Introduction and Theoretical Foundation

Background and Motivation

Dexterous manipulation—unlike simple gripper-based control—requires precise coordination of finger motions and dynamic hand–object contacts to grasp, reorient, and reposition objects. While recent trackers learn from human hand–object motion [19, 38, 1], they remain largely limited to single-hand tasks. Yet many daily activities demand asymmetric bimanual coordination: one hand stabilizes an object while the other opens, rotates, or repositions it.

Key Challenges

1. Data scarcity: Public datasets with fully tracked hand–object sequences featuring sustained two-hand contact remain limited, constraining the diversity of coordination patterns available for learning.
2. Heterogeneous reference difficulty: A shared policy must accommodate trajectories with changing hand roles and contact timing. Hand-pose accuracy alone does not guarantee object manipulation, while requiring both hands to meet contact conditions can mask one hand's progress.
3. Advantage offsets: Unequal clip difficulty creates reference-dependent advantage biases that can weaken policy updates for useful actions in harder sequences.

Theoretical Foundation

The work builds on prior reference-tracking frameworks (DexTrack [19], Dexplore [38]) that combine proximal policy optimization with behavioral cloning, but extends them to the bimanual setting. The key theoretical insight is that successful bimanual tracking requires:

• Object-motion feedback grounded in physical outcomes, not just hand-pose imitation
• Contact-role supervision that respects each hand's reference-defined engagement timing
• Clip-normalized advantages that remove spurious offsets while preserving signal differences within each reference

---

Methodology

Task Definition

Given a reference for two hands and an object, one policy controls both robot hands to reproduce the object motion. A clip spanning $T$ control steps is written as:

$$\boldsymbol{\tau} = \left\{(\hat{\pmb{q}}_{t}^{L}, \hat{\pmb{q}}_{t}^{R}, \hat{\pmb{p}}_{t}^{o}, \hat{\pmb{R}}_{t}^{o})\right\}_{t=0}^{T}$$

where $\hat{\pmb{q}}_{t}^{h} \in \mathbb{R}^{d}$ contains wrist coordinates and finger-joint angles for hand $h \in \{L, R\}$, $\hat{\pmb{p}}_{t}^{o} \in \mathbb{R}^{3}$ is the object position, and $\hat{\pmb{R}}_{t}^{o} \in SO(3)$ is the object orientation.

Control Interface

The policy predicts bounded corrections $\pmb{a}_{t} \in [-1, 1]^{2d}$ to the next reference, producing a PD control target:

$$\pmb{q}_{t}^{\mathrm{pd}} = \mathrm{clamp}(\hat{\pmb{q}}_{t+1} + \pmb{s} \odot \pmb{a}_{t}, \pmb{q}_{\min}, \pmb{q}_{\max})$$

where $\pmb{s} \in \mathbb{R}^{2d}$ sets the correction range and $\odot$ denotes element-wise multiplication.

Object-Centered Per-Hand Reward

Object tracking term:

$$r_{t}^{\mathrm{obj}} = e^{-\kappa_{p}\|\pmb{p}_{t}^{o} - \hat{\pmb{p}}_{t}^{o}\|_{2}} \cdot e^{-\kappa_{R}\angle(\pmb{R}_{t}^{o}, \hat{\pmb{R}}_{t}^{o})} \cdot e^{-\kappa_{\nu}\|\pmb{v}_{t}^{o} - \hat{\pmb{v}}_{t}^{o}\|_{2}}$$

with sensitivity coefficients $\kappa_{p}, \kappa_{R}, \kappa_{\nu} > 0$.

Contact estimation for fingertip $i$ on hand $h$:

$$\alpha_{i,t}^{h} = \mathbf{1}[f_{i,t}^{h} > f_{\min}] \cdot \mathbf{1}[\delta_{i,t}^{h} < \delta_{\max}]$$

where $f_{i,t}^{h}$ is net contact force and $\delta_{i,t}^{h}$ is distance to object surface.

Per-hand contact and hold rewards (averaged over required fingertips $\mathcal{C}_{t}^{h}$):

$$r_{t}^{\mathrm{cont},h} = \frac{1}{|\mathcal{C}_{t}^{h}|}\sum_{i \in \mathcal{C}_{t}^{h}}\left[\zeta e^{-\kappa_{c}\delta_{i,t}^{h}} + (1-\zeta)\alpha_{i,t}^{h}\right], \quad r_{t}^{\mathrm{hold},h} = \frac{1}{|\mathcal{C}_{t}^{h}|}\sum_{i \in \mathcal{C}_{t}^{h}}\alpha_{i,t}^{h}$$

These are aggregated over the engaged-hand set $\mathcal{E}_{t}$ with hand weights $w_{L} \in (0,1]$ and $w_{R} = 1$.

Total reward:

$$r_{t} = (1 + w_{L})\left[\beta_{\mathrm{hand}} r_{t}^{\mathrm{hand}} + \beta_{\mathrm{obj}} r_{t}^{\mathrm{obj}} + \beta_{\mathrm{cont}} r_{t}^{\mathrm{cont}} + \beta_{\mathrm{hold}} r_{t}^{\mathrm{hold}}\right]$$

Clip-Centered Advantages (CCA)

After computing standard GAE advantages $A_{j}^{\mathrm{GAE}}$, transitions are grouped by reference clip $\mathcal{G}_{m} = \{j \in \mathcal{B}: m(j) = m\}$, and the adjusted advantage is:

$$\widetilde{A}_{j} = A_{j}^{\mathrm{GAE}} - \frac{1}{|\mathcal{G}_{m(j)}|}\sum_{k \in \mathcal{G}_{m(j)}} A_{k}^{\mathrm{GAE}}$$

This removes clip-dependent offsets while preserving pairwise differences among transitions within each clip.

Training Objective

Combined PPO + BC actor loss:

$$\mathcal{L}_{\mathrm{actor}}(\theta) = \mathcal{L}_{\mathrm{PPO}}(\theta; \widetilde{A}) + \lambda \mathbb{E}_{(\boldsymbol{o}^{D}, \boldsymbol{a}^{D}) \sim \mathcal{D}}\left\|\mu_{\theta}(\boldsymbol{o}^{D}) - \boldsymbol{a}^{D}\right\|_{2}^{2}$$

BiHOI Dataset

• 32 calibrated RGB cameras at 1920×1232 resolution, 30 fps
• 1,575 sequences, 229 objects, 14.77 hours of unique interaction
• Pipeline: SAM3 masks → multi-view Depth Anything 3 → MV-SAM3D canonical mesh → FoundationPose 6-DoF tracking → WiLoR hand observations → MANO parametric fitting → retargeting to Wuji robot hands

---

Empirical Validation / Results

Main Results (Table 2)

Key findings: DONIC improves SR from 9.43% (Dexplore-bi) to 86.79% on unseen trajectories and from 11.59% to 57.32% on unseen objects. Wrist translation error decreases from 21.88 cm to 3.96 cm (unseen trajectories) and from 19.46 cm to 4.58 cm (unseen objects).

Ablation Findings

• Removing CCA reduces SR by 5.66 points (unseen trajectories) and 9.15 points (unseen objects)
• Removing contact gate reduces SR to 81.13% and 48.78% on the two splits, respectively
• Adding BiHOI data (1,845 → 3,420 clips) improves all seven metrics on both splits, with unseen-object translation error falling by 11.3%

---

Theoretical and Practical Implications

Theoretical Contributions

1. Object-centered per-hand reward design: Demonstrates that successful bimanual tracking requires decoupling object-motion feedback from hand-specific contact supervision, accommodating asynchronous engagement and complementary roles.

2. Clip-Centered Advantages: Introduces a principled method for handling heterogeneous reference difficulty in shared-policy training. By subtracting per-clip advantage means, the approach removes spurious offsets while preserving the relative information needed for policy improvement.

3. Generalist bimanual control: Establishes that a single policy can learn coordinated manipulation across diverse operation categories and human-motion sources, challenging the prevailing approach of training separate specialist policies.

Practical Contributions

1. BiHOI dataset: Provides the community with a large-scale, multi-view bimanual interaction dataset emphasizing sustained two-hand contact, with reconstructed robot-ready references.

2. Data harmonization pipeline: The reconstruction–retargeting–screening pipeline converts heterogeneous human motion captures into a unified reference pool, enabling joint training across datasets.

3. Practical generalization: One checkpoint generalizes zero-shot to unseen trajectories and objects, substantially reducing the engineering burden of per-task fine-tuning.

---

Conclusion

Main Takeaways

• DONIC is the first generalist bimanual reference tracker jointly trained across operation categories and human-motion datasets, requiring no fine-tuning at inference.
• The combination of object-centered per-hand rewards and Clip-Centered Advantages provides complementary learning signals that enable stable joint PPO+BC training.
• BiHOI substantially enriches the bimanual training data landscape, ranking first on all unseen-object metrics when added to training.

Limitations

• Quantitative evaluation focuses on simulated single-object interactions
• Relies on reconstructed, retargeted motion references that may carry human-kinematics biases
• Limited to quasi-static grasping and manipulation; no physical robot deployment

Future Directions

• Extend DONIC to longer-horizon and multi-object manipulation tasks
• Evaluate on an independent benchmark for fair cross-method comparison
• Investigate transfer to physical bimanual systems (sim-to-real)
• Develop richer contact-role annotations that capture dynamic hand–object contact evolution