Summary (Overview)

• HumanCLAW is a novel evaluation framework that decouples a VLM’s action decision (high-level skill selection) from low-level motor execution (balance, tracking), enabling the measurement of pure action intelligence.
• The framework harnesses an off-the-shelf VLM to issue atomic whole-body skill commands (e.g., walk, turn, sit) at sub-second intervals; each command is realized as continuous, physically-realistic full-body motion via a skill-conditioned diffusion transformer and a half-physics simulator.
• HumanCLAW-Bench comprises 1,218 egocentric find‑navigate‑interact episodes across 41 indoor scenes, with progressive success metrics that require both objective completion and subjective acknowledgment.
• Evaluations of nine frontier VLMs reveal that none solves the benchmark; the best model (Gemini‑3.1) achieves only 16.8% full-task success.
• The primary bottleneck is embodied self‑awareness: VLMs lose track of their own body—failing to know where it is, whether it has reached the goal, or whether it has collided with an obstacle—rather than failing at visual perception.

Introduction and Theoretical Foundation

The paper argues that evaluating whether a vision‑language model can act through a physical body requires decoupling action decision from motor control. Existing approaches either:

• Abstract away motion (e.g., code agents, scripted animations) – the model never faces the physical consequences of its actions.
• Entangle decision with control (e.g., end‑to‑end VLA policies) – failures can come from balance loss or motor tracking errors, not from poor reasoning.

The authors introduce action intelligence as the ability to decide, moment by moment, what a physical body should do next inside the execution loop, converting spatial understanding into closed‑loop embodied decisions. They argue that this is the operational component of spatial intelligence and that current VLMs, trained on visual question answering, lack it.

Methodology

1. HumanCLAW Framework (Figure 2)

The closed‑loop system works as follows at each step $t$:

1. The VLM receives an egocentric RGB observation $o_t$, a compact textual history $h_t$, and the high‑level instruction $p$.
2. It outputs a parameterized skill call $\langle s_t, c_t \rangle = \mathcal{H}(p, o_t, h_t)$.
3. A skill‑conditioned motion generator $G$ produces a 0.5‑second motion chunk $\hat{x}^{\text{fut}}_t = G(x_t, s_t, c_t)$.
4. A half‑physics simulator $W$ executes the motion under gravity and collisions, returning the next observation $o_{t+1} = W(x_t, \hat{x}^{\text{fut}}_t)$.

Atomic Skill Set includes: walk, side_step, step_back, turn_in_place, climb_upstairs, walk_downstairs, sit_in_place, stop. Each skill has a deterministic JSON interface and is realized as continuous human motion.

Harness Design (Figure 3): The VLM is guided through:

• Contextual prompting – agent identity, task, available skills, rules.
• Ego‑perception & spatial understanding – explicit description of visible objects, distances, target visibility, obstacles.
• High‑to‑mid‑to‑low reasoning – progressive reasoning from high‑level intent → mid‑level objective (e.g., “approach sofa”) → low‑level skill.
• Skill‑specific verifier – checks spatial safety (e.g., walking into obstacle, sitting without a seat) before execution.

2. Skill‑Conditioned Motion Generator (Figure 4)

• Motion Base DiT: A 10‑layer diffusion transformer trained on AMASS motions for receding‑horizon continuation. It observes 5 history frames and predicts 15 future frames (0.5 s). The motion is represented as tokens: $$u_\tau = [x_{t-4:t}, \tilde{x}_{t+1:t+15}(\tau)] \in \mathbb{R}^{20 \times d}, \ d=219.$$ Flow matching objective: $\|\hat{v}_\theta - v^*\|_2^2$, where $v^* = x^{\text{fut}} - (1 - \sigma_{\min})\epsilon$.
• Plug‑and‑Play Skill ControlNet: Each atomic skill has a separate ControlNet adapter attached to the frozen base DiT. The output is: $$z^{\ell+1} = B^\ell(z^\ell, \tau) + W^\ell C^\ell(z^\ell + e_s(c_t), \tau),$$ where $B^\ell$ is the frozen base block, $C^\ell$ the trainable control block, $e_s$ the skill‑condition encoder, and $W^\ell$ zero‑initialized. New skills are added without retraining existing ones.

3. Half‑Physics Simulation

The world follows rigid‑body physics (collision, friction, gravity) but the human body is driven by kinematic velocities derived from the motion chunk, not by joint torques. This preserves physical interaction (e.g., walls block, objects move) while abstracting balance and motor‑tracking failures. The simulator runs at 120 Hz with passive joint stiffness $\lambda = 1.0$.

4. HumanCLAW‑Bench

• 3D Scenes: 41 HSSD validation houses, 1,218 episodes across six target categories (chair, bed, couch, potted_plant, toilet, tv). 597 episodes require full find‑navigate‑interact (sit); 621 stop at navigation.
• Difficulty Stratification: Episodes are split into Easy/Medium/Hard based on three geometric dimensions: geodesic distance, navigational choice (turns + rooms), and obstacle density (Figure 7).
• Evaluation Metrics:
  • Progressive Success Rate: FindSR (target rendered ≥100 pixels & acknowledged), NavSR (agent stops within 20 cm of target), InteractSR (pelvis contacts target mesh).
  • Action Quality: Motion jerk (third‑order derivative of pelvis trajectory).
  • Body Awareness & Disturbance: Collision fraction (Coll.), number of displaced objects (#Dtb), displacement distance (dDtb).
  • Cost: Input/output tokens per step.

Empirical Validation / Results

Zero‑Shot Motion Fidelity

Table 1: Skill Functional Fidelity. Achievement ratio = achieved / commanded (mean ± std). Our controllable prior tracks every commanded magnitude near unity with small variance; MoMask (text‑to‑motion baseline) ignores magnitude.

Full Benchmark Results

Table 2: Full Validation Results on HumanCLAW‑Bench. Best / second best per column.

Finding 1: No current VLM solves the benchmark. The best model (Gemini‑3.1) achieves only 16.8% InteractSR. The sharpest drop is between FindSR and NavSR (landing rate ~0.65 for Gemini‑3.1). Finding 2: Open model Gemma‑4‑31B is close to proprietary frontiers (11.1% InteractSR).

Ablation Study (Table 3 on mini‑val)

• Verifier is critical: removing it collapses NavSR from 27.0% to 2.0% and InteractSR from 18.9% to 0.0%.
• Text history necessary but saturates: no history → NavSR 11.0%, InteractSR 0.0%; longer history (20–100 steps) does not improve.
• More visual frames can hurt: 10 frames drop NavSR from 27.0% to 13.0%.
• Mid‑level reasoning essential for interaction: removing it collapses InteractSR from 18.9% to 0.0%, even though FindSR and NavSR remain high.

Success‑Rate Variants (Table 4)

Separates objective (geometry‑only) from model‑acknowledged success.

Finding 3: Perception is not the bottleneck – GeoFindSR vs. FindSR differ by only 5–10 points for strong models. The real gap is in bringing the target into view (exploration) and in everything after seeing.

Error Analysis (Figure 8)

• Navigation failures (68% of episodes where target is found): 34% body awareness (arrived but not acknowledged, or jammed), 30% ego‑spatial distance hallucination (stop while far), 17% mid‑level reasoning (lost target), 9% perception, 10% ineffective actions.
• Interaction failures (71% of episodes where navigation succeeded): 81% body awareness errors – 58% sit into thin air, 14% sit on wrong object, 9% sit then stand and move away. Only 19% are decision failures (never issued sit).
• Collision breakdown (Table 5): Legs/feet collide most (28–45%), arms/hands next (20–35%), head rarely (<7%).

Finding 4: Egocentric self‑localization is the navigation bottleneck. Finding 5: Reaching is not interacting – the agent cannot place its body precisely. Finding 6: No current VLM perceives its own body – collisions concentrate in body parts it does not watch.

Theoretical and Practical Implications

• Theoretical: The paper introduces action intelligence as a distinct capability from perception or motor control, operationalized as closed‑loop whole‑body decision‑making. It demonstrates that current VLMs, despite strong visual recognition, fail at the simplest form of embodied reasoning: tracking their own body state.
• Practical: HumanCLAW provides a clean testbed for evaluating embodied reasoning without confounding motor execution. The plug‑and‑play skill design means improvements in reasoning (e.g., better spatial memory, termination detection) transfer directly to the body. The benchmark reveals that the bottleneck is not perception but embodied self‑awareness – a capability that can potentially be improved through better reasoning scaffolds rather than more action data.

Conclusion

HumanCLAW isolates a middle layer of physical action – continuous whole‑body motion with real scene consequences, but without balance or motor‑tracking failures – and uses it to evaluate how well off‑the‑shelf VLMs act through a body. Across nine frontier models, none solves the benchmark; the best achieves only 16.8% full‑task success. The failures are concentrated in embodied self‑awareness: knowing where the body is, whether it has arrived, and what its motion has just caused. With motor execution factored out, these are decision‑level failures, not perception or control failures. The paper argues that generalizable action intelligence arises from reasoning, not from fitting action data, and that future advances should focus on building persistent spatial memory, calibrated termination, and an internal model of the body and its spatial relations. The framework leaves ample headroom and offers a clean testbed for the next generation of agents that not only see and plan, but act.