# Reflex: Visually Grounded Reactive Humanoid Control

> Reflex enables a real humanoid to catch human-thrown boxes from egocentric RGB-D vision, achieving 85.7% simulation success and 65% real-robot success at 2.5 meters.

- **Source:** [arXiv](https://arxiv.org/abs/2610.12772)
- **Published:** 2026-10-13
- **Permalink:** https://picx.dev/p/pMwz2q
- **Whiteboard:** https://picx.dev/p/pMwz2q/image

## Summary

## Summary (Overview)

- **Reflex** is a three-stage framework for visual reactive whole-body catching on a humanoid robot (Unitree G1), enabling it to catch human-thrown boxes using only onboard egocentric RGB-D vision.
- The framework decomposes the task into three learnable capabilities: (1) whole-body control via reinforcement learning, (2) object dynamics inference via a compact latent representation, and (3) visual grounding of that latent from egocentric RGB-D history.
- In simulation, Reflex achieves **85.7% catch success** from egocentric RGB-D, compared to **87.9%** with privileged box observations.
- On the real Unitree G1, Reflex catches human-thrown boxes with **65% and 40% success** at 2.5 m and 5 m, reaching **76% and 59%** of human catcher success rates at the same distances.
- Temporal visual history is critical: removing it reduces success by **35.6 and 39.8 percentage points** on centered and laterally offset throws.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Humans routinely react to fast-moving objects with their entire body—catching a falling cup, stepping aside from a passing cart, or catching a box tossed during truck unloading. These behaviors require:
1. **Whole-body control**: coordinating feet, torso, and arms while maintaining balance
2. **Fast prediction of object dynamics** under latency (the box reaches the robot ~1 second after release, while camera-to-control latency has a median of 69 ms)
3. **Egocentric vision grounding**: the camera moves with the robot, so observed image motion couples object motion with self-motion, and the robot's own arms increasingly occlude the object near contact

### Key Challenges

The task of catching a thrown box is more demanding than intercepting a ball with hands: the robot must reposition its body, use forearms and torso to form a support, absorb momentum, and retain the load without losing balance. The task is difficult even for humans—across trials, three human catchers succeeded on only **28.3%** of throws from 7 m.

### Limitations of Existing Work

Existing systems address only parts of the problem:
- Robotic manipulators catch from fixed/wheeled bases, avoiding whole-body balance challenges
- Recent humanoid systems (goalkeeping, basketball catching) rely on motion capture or retroreflective objects rather than grounding fast object motion from onboard vision
- No existing system combines whole-body control, fast prediction of object dynamics, and egocentric visual grounding on a real humanoid

---

## Methodology

### Three-Stage Decomposition

The deployed policy is decomposed into a visual perception function $V_{\psi}$ and a whole-body control function $\pi_{\theta}$. Given a history of K egocentric RGB-D observations and camera ego-motion, the perception function predicts an 8-dimensional object-dynamics latent:

$$
\widehat{\mathbf{z}}_{t} = V_{\psi}\left(\mathbf{I}_{t-K+1:t}, \mathbf{e}_{t-K+1:t}\right), \quad \mathbf{a}_{t} = \pi_{\theta}\left(\mathbf{r}_{t}, \widehat{\mathbf{z}}_{t}\right).\tag{1}
$$

The action is converted to nominal joint-position targets:

$$
\mathbf{q}_{t}^{\text{nom}} = \mathbf{q}_{0} + \mathbf{s} \odot \mathbf{a}_{t},\tag{2}
$$

where $\mathbf{q}_{0} \in \mathbb{R}^{29}$ is the nominal joint configuration, $\mathbf{s} \in \mathbb{R}^{29}$ contains per-joint action scales, and ⊙ denotes elementwise multiplication.

### Stage 1: Learning Whole-Body Control

- Trains a state-based catching policy $\pi_{\text{state}}$ using PPO with a 170-dimensional observation combining robot/reference input with ten box-state channels
- Uses an asymmetric critic that observes uncorrupted full-body poses and ground-truth box state
- Employs approach shaping, hold shaping, sustained-catch bonus, human reference motion tracking, and adversarial motion prior (AMP)
- Trains in rendering-free simulation for efficiency

### Stage 2: Learning Object Dynamics

The privileged encoder $E_{\phi}$ maps a box observation $\mathbf{p}_{t} \in \mathbb{R}^{26}$ to the latent:

$$
\mathbf{z}_{t} = E_{\phi}(\mathbf{p}_{t}) \in [-1, 1]^{8}.\tag{3}
$$

Key components:
- **Action-matching initialization**: Behavior cloning from the teacher's actions with visibility-based sample weighting (weight 5 for visible-box samples, 1 otherwise)
- **DAgger-style data aggregation**: collecting teacher action labels on states visited under latent control
- **Task-specific PPO refinement**: jointly optimizing the privileged encoder and full decoder
- **Latency randomization**: sampling observation latency from 1–3 control steps (20–60 ms)

### Stage 3: Learning Visual Grounding

The visual encoder uses:
- A CNN predicting a box-localization heatmap
- A fixed geometric layer forming $\mathbf{g}_{t} \in \mathbb{R}^{8}$ from normalized image coordinates, local minimum depths, heatmap peak score, and camera-frame surface point
- A GRU processing these features plus 9-dimensional ego-motion vectors

The temporal training objective is:

$$
\begin{array}{c} \mathcal{L}_{V} = \mathbb{E}_{\mathcal{D}_{V}} \left[ \omega_{t}\left(\ell_{z}(t) + \lambda_{\text{aux}} \ell_{\text{state}}(t)\right) \right], \\ \ell_{z}(t) = \frac{1}{8} \sum_{j=1}^{8} \alpha_{j}\left(\frac{\widehat{z}_{t,j} - z_{t,j}}{\sigma_{j} + \varepsilon}\right)^{2}, \\ \ell_{\text{state}}(t) = \frac{1}{6} \| \widehat{\mathbf{u}}_{t} - \mathbf{u}_{t} \|_{2}^{2}. \end{array}\tag{5}
$$

Here $\sigma_{j}$ is the training-set standard deviation of latent coordinate $j$, $\alpha_{j}$ are nonnegative weights reflecting the decoder's action sensitivity to latent errors, and $\lambda_{\text{aux}}$ balances auxiliary supervision.

---

## Empirical Validation / Results

### Simulation Performance

**Overall catch success:** Visual controller catches 3290/3840 throws (85.7%), compared to 87.9% with privileged latent input.

### Real-Robot Performance

**Table II: Real-robot catching rates (%) over 20 G1 trials per condition**

| Catcher | Small box | Regular box | Large box | 0.5 kg | 1.0 kg | 2.0 kg | 2.5 m | 5.0 m | 7.0 m |
|---------|-----------|-------------|-----------|--------|--------|--------|-------|-------|-------|
| Real G1 (ours) | 20.0 | 65.0 | 45.0 | 45.0 | 60.0 | 30.0 | 65.0 | 40.0 | 15.0 |
| Human reference | 75.0 | 85.0 | 78.3 | 85.0 | 86.7 | 76.7 | 85.0 | 68.3 | 28.3 |

### Key Ablation Results

**Table V: Visual-encoder ablation (catch rates %, mean ± S.D. over five policies)**

| Visual encoder | Offset 0.0 m | Offset 0.6 m |
|----------------|--------------|--------------|
| Privileged latent z (reference) | 87.9±3.5 | 74.3±4.5 |
| **Full visual encoder** | **85.2±4.2** | **66.6±8.9** |
| w/o camera ego-motion | 74.8±3.8 | 44.0±3.2 |
| w/o temporal history (single frame) | 49.6±4.9 | 26.8±2.4 |
| w/o DAgger | 86.2±2.8 | 67.6±4.4 |
| w/o box-state loss | 85.0±3.8 | 70.3±4.1 |
| w/o heatmap supervision | 77.8±5.5 | 55.7±7.0 |
| w/o latent regression (state regression) | 66.5±7.1 | 47.1±2.8 |

### Latency Sensitivity

Hardware measurements show camera-to-control latency has p50 = 69.1 ms, p95 = 90.2 ms. With visual input, catch success degrades gracefully: 88.4% at 0 ms added delay, 87.2% at 20 ms, 83.0% at 40 ms, 72.7% at 60 ms, and 60.9% at 80 ms. Latency randomization during training helps by 5–11 points at every tested delay.

### Downstream Task

In the 20-trial downstream task evaluation (catch → carry 7.8 m → handover → walk 7.3 m back), cumulative success rates are:
- 65% after catching
- 35% after carrying
- 30% after handover
- 25% after return

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Decomposition principle**: The three-stage framework demonstrates that complex visuomotor skills can be decomposed into separable capabilities (control, dynamics inference, visual grounding) connected by a compact latent representation, with each stage receiving direct training signal.

2. **Latent interface design**: The 8-dimensional object-dynamics latent (learned through action matching and task optimization) outperforms regressing the full 26-dimensional privileged state (85.7% vs. 66.5% catch success), showing that task-relevant compression improves visual learning.

3. **Efficiency**: Decoupling visual learning from reinforcement learning reduces computational cost dramatically—Stage 2 PPO runs at ~28k environment steps/s (rendering-free) vs. ~1.5k steps/s with rendering, reducing 59M steps from ~11 hours to ~35 minutes.

### Practical Implications

1. **Real-world viability**: Reflex demonstrates that humanoids can perform reactive whole-body tasks from onboard vision alone, without motion capture or instrumented environments.

2. **Generalization**: The framework generalizes across box sizes, weights (0.5–2 kg), colors (7 tested), and throw distances (2.5–7 m) in simulation, with real-robot success across all tested conditions.

3. **Downstream integration**: The catching controller connects seamlessly to existing locomotion controllers (SONIC) for carrying and handover, showing modular composability.

---

## Conclusion

Reflex learns visual reactive whole-body catching on a humanoid by decoupling whole-body control, object dynamics, and visual grounding into three stages linked by an object-dynamics latent. In simulation, visual control reaches 85.7% catch success versus 87.9% with privileged input, and on a real Unitree G1 it catches human-thrown boxes at 76% and 59% of human success at 2.5 and 5 m.

### Key Limitations

- **Long-range visual grounding**: On 5 m throws, visual success falls to 43.0% while privileged input reaches 79.7%; real-robot success falls to 40% and 15% at 5 and 7 m
- **Size blindness**: The controller is effectively size-blind, with performance dropping for boxes outside the reference size
- **Single task focus**: Evaluated on one task with human throws and 20 real trials per condition
- **Rendering gaps**: Motion blur, auto-exposure dynamics, and structured depth artifacts are not modeled

### Future Directions

Extending the decomposition to other visual reactive whole-body tasks—interception, avoidance, handoffs, and physical assistance—is a natural next step. These everyday tasks require predicting how the world will change and coordinating the body before contact.

---

_Markdown view of https://picx.dev/p/pMwz2q, served by PicX — AI-generated visual whiteboard summaries of research papers._