Summary (Overview)

• Reflex is a three-stage framework for visual reactive whole-body catching on a humanoid robot (Unitree G1), enabling it to catch human-thrown boxes using only onboard egocentric RGB-D vision.
• The framework decomposes the task into three learnable capabilities: (1) whole-body control via reinforcement learning, (2) object dynamics inference via a compact latent representation, and (3) visual grounding of that latent from egocentric RGB-D history.
• In simulation, Reflex achieves 85.7% catch success from egocentric RGB-D, compared to 87.9% with privileged box observations.
• On the real Unitree G1, Reflex catches human-thrown boxes with 65% and 40% success at 2.5 m and 5 m, reaching 76% and 59% of human catcher success rates at the same distances.
• Temporal visual history is critical: removing it reduces success by 35.6 and 39.8 percentage points on centered and laterally offset throws.

---

Introduction and Theoretical Foundation

Background and Motivation

Humans routinely react to fast-moving objects with their entire body—catching a falling cup, stepping aside from a passing cart, or catching a box tossed during truck unloading. These behaviors require:

1. Whole-body control: coordinating feet, torso, and arms while maintaining balance
2. Fast prediction of object dynamics under latency (the box reaches the robot ~1 second after release, while camera-to-control latency has a median of 69 ms)
3. Egocentric vision grounding: the camera moves with the robot, so observed image motion couples object motion with self-motion, and the robot's own arms increasingly occlude the object near contact

Key Challenges

The task of catching a thrown box is more demanding than intercepting a ball with hands: the robot must reposition its body, use forearms and torso to form a support, absorb momentum, and retain the load without losing balance. The task is difficult even for humans—across trials, three human catchers succeeded on only 28.3% of throws from 7 m.

Limitations of Existing Work

Existing systems address only parts of the problem:

• Robotic manipulators catch from fixed/wheeled bases, avoiding whole-body balance challenges
• Recent humanoid systems (goalkeeping, basketball catching) rely on motion capture or retroreflective objects rather than grounding fast object motion from onboard vision
• No existing system combines whole-body control, fast prediction of object dynamics, and egocentric visual grounding on a real humanoid

---

Methodology

Three-Stage Decomposition

The deployed policy is decomposed into a visual perception function $V_{\psi}$ and a whole-body control function $\pi_{\theta}$. Given a history of K egocentric RGB-D observations and camera ego-motion, the perception function predicts an 8-dimensional object-dynamics latent:

$$\widehat{\mathbf{z}}_{t} = V_{\psi}\left(\mathbf{I}_{t-K+1:t}, \mathbf{e}_{t-K+1:t}\right), \quad \mathbf{a}_{t} = \pi_{\theta}\left(\mathbf{r}_{t}, \widehat{\mathbf{z}}_{t}\right).\tag{1}$$

The action is converted to nominal joint-position targets:

$$\mathbf{q}_{t}^{\text{nom}} = \mathbf{q}_{0} + \mathbf{s} \odot \mathbf{a}_{t},\tag{2}$$

where $\mathbf{q}_{0} \in \mathbb{R}^{29}$ is the nominal joint configuration, $\mathbf{s} \in \mathbb{R}^{29}$ contains per-joint action scales, and ⊙ denotes elementwise multiplication.

Stage 1: Learning Whole-Body Control

• Trains a state-based catching policy $\pi_{\text{state}}$ using PPO with a 170-dimensional observation combining robot/reference input with ten box-state channels
• Uses an asymmetric critic that observes uncorrupted full-body poses and ground-truth box state
• Employs approach shaping, hold shaping, sustained-catch bonus, human reference motion tracking, and adversarial motion prior (AMP)
• Trains in rendering-free simulation for efficiency

Stage 2: Learning Object Dynamics

The privileged encoder $E_{\phi}$ maps a box observation $\mathbf{p}_{t} \in \mathbb{R}^{26}$ to the latent:

$$\mathbf{z}_{t} = E_{\phi}(\mathbf{p}_{t}) \in [-1, 1]^{8}.\tag{3}$$

Key components:

• Action-matching initialization: Behavior cloning from the teacher's actions with visibility-based sample weighting (weight 5 for visible-box samples, 1 otherwise)
• DAgger-style data aggregation: collecting teacher action labels on states visited under latent control
• Task-specific PPO refinement: jointly optimizing the privileged encoder and full decoder
• Latency randomization: sampling observation latency from 1–3 control steps (20–60 ms)

Stage 3: Learning Visual Grounding

The visual encoder uses:

• A CNN predicting a box-localization heatmap
• A fixed geometric layer forming $\mathbf{g}_{t} \in \mathbb{R}^{8}$ from normalized image coordinates, local minimum depths, heatmap peak score, and camera-frame surface point
• A GRU processing these features plus 9-dimensional ego-motion vectors

The temporal training objective is:

$$\begin{array}{c} \mathcal{L}_{V} = \mathbb{E}_{\mathcal{D}_{V}} \left[ \omega_{t}\left(\ell_{z}(t) + \lambda_{\text{aux}} \ell_{\text{state}}(t)\right) \right], \\ \ell_{z}(t) = \frac{1}{8} \sum_{j=1}^{8} \alpha_{j}\left(\frac{\widehat{z}_{t,j} - z_{t,j}}{\sigma_{j} + \varepsilon}\right)^{2}, \\ \ell_{\text{state}}(t) = \frac{1}{6} \| \widehat{\mathbf{u}}_{t} - \mathbf{u}_{t} \|_{2}^{2}. \end{array}\tag{5}$$

Here $\sigma_{j}$ is the training-set standard deviation of latent coordinate $j$, $\alpha_{j}$ are nonnegative weights reflecting the decoder's action sensitivity to latent errors, and $\lambda_{\text{aux}}$ balances auxiliary supervision.

---

Empirical Validation / Results

Simulation Performance

Overall catch success: Visual controller catches 3290/3840 throws (85.7%), compared to 87.9% with privileged latent input.

Real-Robot Performance

Table II: Real-robot catching rates (%) over 20 G1 trials per condition

Key Ablation Results

Table V: Visual-encoder ablation (catch rates %, mean ± S.D. over five policies)

Latency Sensitivity

Hardware measurements show camera-to-control latency has p50 = 69.1 ms, p95 = 90.2 ms. With visual input, catch success degrades gracefully: 88.4% at 0 ms added delay, 87.2% at 20 ms, 83.0% at 40 ms, 72.7% at 60 ms, and 60.9% at 80 ms. Latency randomization during training helps by 5–11 points at every tested delay.

Downstream Task

In the 20-trial downstream task evaluation (catch → carry 7.8 m → handover → walk 7.3 m back), cumulative success rates are:

• 65% after catching
• 35% after carrying
• 30% after handover
• 25% after return

---

Theoretical and Practical Implications

Theoretical Contributions

1. Decomposition principle: The three-stage framework demonstrates that complex visuomotor skills can be decomposed into separable capabilities (control, dynamics inference, visual grounding) connected by a compact latent representation, with each stage receiving direct training signal.

2. Latent interface design: The 8-dimensional object-dynamics latent (learned through action matching and task optimization) outperforms regressing the full 26-dimensional privileged state (85.7% vs. 66.5% catch success), showing that task-relevant compression improves visual learning.

3. Efficiency: Decoupling visual learning from reinforcement learning reduces computational cost dramatically—Stage 2 PPO runs at ~28k environment steps/s (rendering-free) vs. ~1.5k steps/s with rendering, reducing 59M steps from ~11 hours to ~35 minutes.

Practical Implications

1. Real-world viability: Reflex demonstrates that humanoids can perform reactive whole-body tasks from onboard vision alone, without motion capture or instrumented environments.

2. Generalization: The framework generalizes across box sizes, weights (0.5–2 kg), colors (7 tested), and throw distances (2.5–7 m) in simulation, with real-robot success across all tested conditions.

3. Downstream integration: The catching controller connects seamlessly to existing locomotion controllers (SONIC) for carrying and handover, showing modular composability.

---

Conclusion

Reflex learns visual reactive whole-body catching on a humanoid by decoupling whole-body control, object dynamics, and visual grounding into three stages linked by an object-dynamics latent. In simulation, visual control reaches 85.7% catch success versus 87.9% with privileged input, and on a real Unitree G1 it catches human-thrown boxes at 76% and 59% of human success at 2.5 and 5 m.

Key Limitations

• Long-range visual grounding: On 5 m throws, visual success falls to 43.0% while privileged input reaches 79.7%; real-robot success falls to 40% and 15% at 5 and 7 m
• Size blindness: The controller is effectively size-blind, with performance dropping for boxes outside the reference size
• Single task focus: Evaluated on one task with human throws and 20 real trials per condition
• Rendering gaps: Motion blur, auto-exposure dynamics, and structured depth artifacts are not modeled

Future Directions

Extending the decomposition to other visual reactive whole-body tasks—interception, avoidance, handoffs, and physical assistance—is a natural next step. These everyday tasks require predicting how the world will change and coordinating the body before contact.