# DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations

> DONIC is the first generalist bimanual reference tracker using object-centered per-hand rewards and clip-centered advantages, achieving 86.79% success on unseen trajectories.

- **Source:** [arXiv](https://arxiv.org/abs/2610.29997)
- **Published:** 2026-10-23
- **Permalink:** https://picx.dev/p/XiSC0u
- **Whiteboard:** https://picx.dev/p/XiSC0u/image

## Summary

## Summary (Overview)

- **DONIC** is the first generalist bimanual reference tracker that jointly trains across operation categories and human-motion sources, achieving state-of-the-art performance on unseen trajectories and objects without fine-tuning.
- **BiHOI** is a new multi-view hand–object interaction dataset with 1,575 sequences spanning 229 objects and 14.77 hours, emphasizing sustained bimanual contact (65.91% both-hand contact rate, exceeding prior datasets by 27+ percentage points).
- The method combines **proximal policy optimization (PPO)** with **behavioral cloning (BC)** in an iterative feedback loop, using an **object-centered per-hand reward** to handle asymmetric bimanual coordination.
- A novel **Clip-Centered Advantages (CCA)** mechanism removes reference-dependent advantage offsets while preserving within-trajectory differences, enabling stable joint training across heterogeneous references.
- A single checkpoint achieves **86.79% success** on unseen trajectories and **57.32%** on held-out objects, surpassing the strongest bimanual baseline by **77.36 and 45.73 percentage points**, respectively.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Dexterous manipulation—unlike simple gripper-based control—requires precise coordination of finger motions and dynamic hand–object contacts to grasp, reorient, and reposition objects. While recent trackers learn from human hand–object motion [19, 38, 1], they remain largely limited to **single-hand tasks**. Yet many daily activities demand **asymmetric bimanual coordination**: one hand stabilizes an object while the other opens, rotates, or repositions it.

### Key Challenges

1. **Data scarcity**: Public datasets with fully tracked hand–object sequences featuring sustained two-hand contact remain limited, constraining the diversity of coordination patterns available for learning.
2. **Heterogeneous reference difficulty**: A shared policy must accommodate trajectories with changing hand roles and contact timing. Hand-pose accuracy alone does not guarantee object manipulation, while requiring both hands to meet contact conditions can mask one hand's progress.
3. **Advantage offsets**: Unequal clip difficulty creates reference-dependent advantage biases that can weaken policy updates for useful actions in harder sequences.

### Theoretical Foundation

The work builds on prior reference-tracking frameworks (DexTrack [19], Dexplore [38]) that combine **proximal policy optimization** with **behavioral cloning**, but extends them to the bimanual setting. The key theoretical insight is that successful bimanual tracking requires:
- **Object-motion feedback** grounded in physical outcomes, not just hand-pose imitation
- **Contact-role supervision** that respects each hand's reference-defined engagement timing
- **Clip-normalized advantages** that remove spurious offsets while preserving signal differences within each reference

---

## Methodology

### Task Definition

Given a reference for two hands and an object, one policy controls both robot hands to reproduce the object motion. A clip spanning $T$ control steps is written as:

$$
\boldsymbol{\tau} = \left\{(\hat{\pmb{q}}_{t}^{L}, \hat{\pmb{q}}_{t}^{R}, \hat{\pmb{p}}_{t}^{o}, \hat{\pmb{R}}_{t}^{o})\right\}_{t=0}^{T}
$$

where $\hat{\pmb{q}}_{t}^{h} \in \mathbb{R}^{d}$ contains wrist coordinates and finger-joint angles for hand $h \in \{L, R\}$, $\hat{\pmb{p}}_{t}^{o} \in \mathbb{R}^{3}$ is the object position, and $\hat{\pmb{R}}_{t}^{o} \in SO(3)$ is the object orientation.

### Control Interface

The policy predicts bounded corrections $\pmb{a}_{t} \in [-1, 1]^{2d}$ to the next reference, producing a **PD control target**:

$$
\pmb{q}_{t}^{\mathrm{pd}} = \mathrm{clamp}(\hat{\pmb{q}}_{t+1} + \pmb{s} \odot \pmb{a}_{t}, \pmb{q}_{\min}, \pmb{q}_{\max})
$$

where $\pmb{s} \in \mathbb{R}^{2d}$ sets the correction range and $\odot$ denotes element-wise multiplication.

### Object-Centered Per-Hand Reward

**Object tracking term:**

$$
r_{t}^{\mathrm{obj}} = e^{-\kappa_{p}\|\pmb{p}_{t}^{o} - \hat{\pmb{p}}_{t}^{o}\|_{2}} \cdot e^{-\kappa_{R}\angle(\pmb{R}_{t}^{o}, \hat{\pmb{R}}_{t}^{o})} \cdot e^{-\kappa_{\nu}\|\pmb{v}_{t}^{o} - \hat{\pmb{v}}_{t}^{o}\|_{2}}
$$

with sensitivity coefficients $\kappa_{p}, \kappa_{R}, \kappa_{\nu} > 0$.

**Contact estimation** for fingertip $i$ on hand $h$:

$$
\alpha_{i,t}^{h} = \mathbf{1}[f_{i,t}^{h} > f_{\min}] \cdot \mathbf{1}[\delta_{i,t}^{h} < \delta_{\max}]
$$

where $f_{i,t}^{h}$ is net contact force and $\delta_{i,t}^{h}$ is distance to object surface.

**Per-hand contact and hold rewards** (averaged over required fingertips $\mathcal{C}_{t}^{h}$):

$$
r_{t}^{\mathrm{cont},h} = \frac{1}{|\mathcal{C}_{t}^{h}|}\sum_{i \in \mathcal{C}_{t}^{h}}\left[\zeta e^{-\kappa_{c}\delta_{i,t}^{h}} + (1-\zeta)\alpha_{i,t}^{h}\right], \quad
r_{t}^{\mathrm{hold},h} = \frac{1}{|\mathcal{C}_{t}^{h}|}\sum_{i \in \mathcal{C}_{t}^{h}}\alpha_{i,t}^{h}
$$

These are aggregated over the engaged-hand set $\mathcal{E}_{t}$ with hand weights $w_{L} \in (0,1]$ and $w_{R} = 1$.

**Total reward:**

$$
r_{t} = (1 + w_{L})\left[\beta_{\mathrm{hand}} r_{t}^{\mathrm{hand}} + \beta_{\mathrm{obj}} r_{t}^{\mathrm{obj}} + \beta_{\mathrm{cont}} r_{t}^{\mathrm{cont}} + \beta_{\mathrm{hold}} r_{t}^{\mathrm{hold}}\right]
$$

### Clip-Centered Advantages (CCA)

After computing standard GAE advantages $A_{j}^{\mathrm{GAE}}$, transitions are grouped by reference clip $\mathcal{G}_{m} = \{j \in \mathcal{B}: m(j) = m\}$, and the adjusted advantage is:

$$
\widetilde{A}_{j} = A_{j}^{\mathrm{GAE}} - \frac{1}{|\mathcal{G}_{m(j)}|}\sum_{k \in \mathcal{G}_{m(j)}} A_{k}^{\mathrm{GAE}}
$$

This removes clip-dependent offsets while preserving pairwise differences among transitions within each clip.

### Training Objective

Combined PPO + BC actor loss:

$$
\mathcal{L}_{\mathrm{actor}}(\theta) = \mathcal{L}_{\mathrm{PPO}}(\theta; \widetilde{A}) + \lambda \mathbb{E}_{(\boldsymbol{o}^{D}, \boldsymbol{a}^{D}) \sim \mathcal{D}}\left\|\mu_{\theta}(\boldsymbol{o}^{D}) - \boldsymbol{a}^{D}\right\|_{2}^{2}
$$

### BiHOI Dataset

- **32 calibrated RGB cameras** at 1920×1232 resolution, 30 fps
- **1,575 sequences**, 229 objects, 14.77 hours of unique interaction
- **Pipeline**: SAM3 masks → multi-view Depth Anything 3 → MV-SAM3D canonical mesh → FoundationPose 6-DoF tracking → WiLoR hand observations → MANO parametric fitting → retargeting to Wuji robot hands

---

## Empirical Validation / Results

### Main Results (Table 2)

| Dataset | Method | $T_o$↓ (cm) | $R_o$↓ (rad) | $T_h$↓ (cm) | $R_h$↓ (rad) | $E_{finger}$↓ (rad) | SR↑(%) | Comp↑(%) |
|---------|--------|-----------|-----------|-----------|-----------|-----------------|--------|----------|
| **Unseen trajectory** | DexTrack-bi | 41.86 | 2.3340 | 23.03 | 0.8296 | 0.6301 | 0.00 | 1.30 |
| | Dexplore-bi | 14.73 | 0.8597 | 21.88 | 0.8412 | 0.3166 | 9.43 | 27.87 |
| | w/o CCA | 11.27 | 0.9585 | 4.34 | 0.2980 | 0.0433 | 81.13 | 62.72 |
| | w/o contact gate | 10.26 | 0.9589 | 4.00 | 0.3013 | 0.0338 | 81.13 | 65.03 |
| | **Ours** | **10.27** | **0.9277** | **3.96** | **0.2905** | **0.0372** | **86.79** | **67.62** |
| | Ours+BiHOI | 9.74 | 0.8637 | 3.32 | 0.2818 | 0.0338 | 88.68 | 68.72 |
| **Unseen object** | DexTrack-bi | 127.87 | 2.3532 | 72.83 | 1.0629 | 0.6455 | 0.00 | 0.88 |
| | Dexplore-bi | 17.92 | 1.1431 | 19.46 | 0.9509 | 0.3685 | 11.59 | 26.37 |
| | w/o CCA | 16.83 | 1.1464 | 5.52 | 0.3235 | 0.0378 | 48.17 | 43.89 |
| | w/o contact gate | 17.61 | 1.2041 | 4.97 | 0.3386 | 0.0371 | 48.78 | 49.52 |
| | **Ours** | **15.74** | **1.1310** | **4.58** | **0.3226** | **0.0373** | **57.32** | **54.69** |
| | Ours+BiHOI | 13.96 | 1.1157 | 4.54 | 0.3206 | 0.0361 | 59.76 | 55.35 |

> **Key findings**: DONIC improves SR from 9.43% (Dexplore-bi) to 86.79% on unseen trajectories and from 11.59% to 57.32% on unseen objects. Wrist translation error decreases from 21.88 cm to 3.96 cm (unseen trajectories) and from 19.46 cm to 4.58 cm (unseen objects).

### Ablation Findings

- **Removing CCA** reduces SR by 5.66 points (unseen trajectories) and 9.15 points (unseen objects)
- **Removing contact gate** reduces SR to 81.13% and 48.78% on the two splits, respectively
- **Adding BiHOI data** (1,845 → 3,420 clips) improves **all seven metrics** on both splits, with unseen-object translation error falling by 11.3%

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Object-centered per-hand reward design**: Demonstrates that successful bimanual tracking requires decoupling object-motion feedback from hand-specific contact supervision, accommodating asynchronous engagement and complementary roles.

2. **Clip-Centered Advantages**: Introduces a principled method for handling heterogeneous reference difficulty in shared-policy training. By subtracting per-clip advantage means, the approach removes spurious offsets while preserving the *relative* information needed for policy improvement.

3. **Generalist bimanual control**: Establishes that a single policy can learn coordinated manipulation across diverse operation categories and human-motion sources, challenging the prevailing approach of training separate specialist policies.

### Practical Contributions

1. **BiHOI dataset**: Provides the community with a large-scale, multi-view bimanual interaction dataset emphasizing sustained two-hand contact, with reconstructed robot-ready references.

2. **Data harmonization pipeline**: The reconstruction–retargeting–screening pipeline converts heterogeneous human motion captures into a unified reference pool, enabling joint training across datasets.

3. **Practical generalization**: One checkpoint generalizes zero-shot to unseen trajectories and objects, substantially reducing the engineering burden of per-task fine-tuning.

---

## Conclusion

### Main Takeaways

- DONIC is the **first generalist bimanual reference tracker** jointly trained across operation categories and human-motion datasets, requiring no fine-tuning at inference.
- The combination of **object-centered per-hand rewards** and **Clip-Centered Advantages** provides complementary learning signals that enable stable joint PPO+BC training.
- BiHOI substantially enriches the bimanual training data landscape, ranking first on all unseen-object metrics when added to training.

### Limitations

- Quantitative evaluation focuses on **simulated single-object interactions**
- Relies on **reconstructed, retargeted motion references** that may carry human-kinematics biases
- Limited to quasi-static grasping and manipulation; no physical robot deployment

### Future Directions

- Extend DONIC to **longer-horizon and multi-object manipulation** tasks
- Evaluate on an **independent benchmark** for fair cross-method comparison
- Investigate **transfer to physical bimanual systems** (sim-to-real)
- Develop richer contact-role annotations that capture dynamic hand–object contact evolution

---

_Markdown view of https://picx.dev/p/XiSC0u, served by PicX — AI-generated visual whiteboard summaries of research papers._