HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents

Summary (Overview)

• HexaShell is a system that enables vision language model (VLM) coding agents to control a dual-arm robot through a thin command-line harness, treating robot manipulation as a coding task where failures become reusable tools.
• The system achieves 39.7% success rate and 48.0 progress score on all 42 RoboDojo tasks, establishing a new state of the art on the leaderboard, surpassing learned policies like VPP2 (32.3%) and PhysicalRSI (31.4%).
• The key technical innovation is using multi-view triangulation from calibrated RGB cameras instead of monocular depth networks, which improves performance by 8.7 points and wins 57 layouts vs. 23 losses when paired by layout.
• The research reveals that the primary bottleneck is physical contact, not reasoning: pushing tasks reach 92.5% success and memory tasks 65.0%, but tight insertion only achieves 8.3%.
• The evolution loop operates with a median repair round of 6.3 minutes, requiring no robot data, training, or model changes—only code edits with unit tests.

Introduction and Theoretical Foundation

Background and Motivation

Robot manipulation leaderboards have traditionally been dominated by learned policies—vision language action models and world action models trained on large demonstration datasets that output robot actions directly. However, VLM-based agents have begun competing, with the base model used in this paper reaching only 22.5% without tools.

The Coding Agent Paradigm

The paper draws inspiration from software development workflows where coding agents:

• Read code and run it
• Read error messages and fix code
• Persist fixes across episodes

HexaShell applies this paradigm to robotics: the agent writes and repairs tools (small programs) that persist across episodes, inverting the traditional division of labor where humans write primitives and agents generate per-instruction programs.

Key Theoretical Challenge: Pixels to Positions

The central difficulty is converting visual observations to precise 3D positions (requiring ~1 cm accuracy for grasps). The paper identifies that:

• VLMs judge metric distances poorly in images
• Monocular depth networks (e.g., Depth Anything V2) have median errors of 41 mm at head camera and ~100 mm at wrist cameras
• Triangulation from calibrated cameras provides a more reliable alternative: a point visible in two cameras lies where their rays meet, and a point on a known plane lies where its ray meets that plane

Methodology

System Architecture

HexaShell consists of three components:

1. The Harness — A thin translation layer that:

• Converts agent commands into joint motion via cuRobo inverse kinematics
• Provides three 640×480 RGB cameras (head + two wrists) with calibration
• Validates commands and returns JSON records with planning verdicts and remaining budget
• Isolates ground truth by architecture (no simulator, task, or scoring code in the agent's container)

2. Tools — Small programs that:

• Add sub-commands to the robo interface
• Use only what the robot observes (no ground truth access)
• Measure 3D positions via three constraint types:
  • Table plane (measured via plane-sweep stereo, ~1 mm accuracy)
  • Known horizontal planes (block tops, bowl rims, slots)
  • Second views (triangulation from two cameras, requiring robot motion)

3. Evolution Loop — The repair cycle:

• Acting agent attempts a layout with current tools
• Optimizer agent (same model) diagnoses failures from logs, camera frames, and transcripts
• Optimizer writes/fixes tools with unit tests
• Delivery check verifies tools load and interface text is clean
• Same layout retried (up to 5 edits per layout)

Key Design Principles

• The harness translates and never decides
• Ground truth is isolated by architecture
• Strategy lives in tools, not prompts
• Every number about an object must be measured in the episode (no benchmark asset data)

Experimental Setup

• Benchmark: RoboDojo (42 simulated tasks, dual-arm ARX X5 in Isaac Sim)
• Task types: Generalization (12 tasks + cluttered variants), Memory, Long-Horizon, Precision, Open
• Model: Codex with GPT-6-Astra at medium reasoning effort (unmodified)
• Training: Tools built on 10 layouts per task (development budget: 12 hours per task)
• Evaluation: Frozen tools tested on 10 new layouts (layouts 10–19 of seed 0)

Empirical Validation / Results

Main Results

Triangulation vs. Depth Network (Table 1)

Capability Map (Table 3)

Cost of Second Views (Table 2)

• 3.6% of executed steps spent on viewing (vs. 0.7% with depth network)
• 43% of episodes contain at least one viewing move
• On layouts with viewing moves: triangulation succeeds 39 times vs. 22 for depth network (p=0.002)
• Viewing costs are concentrated: highest for egg carton (40% of steps) and coin slot (16%)

Clutter Robustness

• HexaShell keeps 95% of success on cluttered layouts (35.0% → 33.3%)
• Learned policies keep only 1–68% (median 18%)
• VPP2 drops from 46.4% to 12.6%; PhysicalRSI from 25.7% to 5.6%

Theoretical and Practical Implications

Where Tools Help vs. Fall Short

The paper provides a capability map showing that success follows the contact a task needs, not reasoning difficulty:

• Strong: Pushing (92.5%), memory tasks (65.0%), rigid manipulation (50.0%)
• Weak: Tight insertion (8.3%), articulated objects (7.5%), deformable materials (33.3%)
• Key insight: Tools add most where the right action can be computed from measured geometry (pushing gains +33 points, rigid +13, memory +26)

Failure Mode Analysis

The paper distinguishes two failure types:

1. Declared successes (30%): Agent believes it succeeded but checker disagrees → perception gap
2. Exhausted budgets (57%): Agent sees failure but cannot fix motion → execution gap

The Depth vs. Geometry Trade-off

• Rendered depth still outperforms triangulation by 10.6 points (48.8 vs. 38.3)
• Most of the gap is in tight insertion (30.1% vs. 8.3%)
• The limitation: insertions need hole and part in the same frame to a few millimeters, but wrist views must be made while the gripper holds the part

Case Study Insights

Seven case studies demonstrate the repair loop:

• Push T: Fixed a height check comparing unequal pixels (area from each pixel's footprint)
• Swap T: Fixed color mask mixing top and side faces (match by hue, keep bright top face)
• Play Xylophone: Added second view for held mallet head (triangulation, 2.7 mm accuracy)
• Conveyor pick: Fixed impossible schedules (estimate motion cost first)
• Tic tac toe: Optimized step budget (turn gripper while moving)
• Fold clothes: Fixed grasp point near edge (require 12 mm of cloth around point)
• Insert tubes: Fixed early stopping (recheck original path from reached joints)

Conclusion

HexaShell demonstrates that an unchanged coding agent can achieve state-of-the-art results on robot manipulation by:

1. Treating robot failures as bugs to fix in persistent tools
2. Using geometric triangulation instead of learned depth estimation
3. Maintaining a thin harness that makes no decisions for the agent

The fundamental limit is contact, not reasoning. The next step identified by the authors is developing tools that:

• Keep measuring while the gripper holds the part
• Correct motion during contact rather than planning entirely in advance

The system is open-sourced at https://github.com/HexaFuture/HexaShell, with 327 task commands in ~78,000 lines of Python evolved through 769 optimizer rounds (487 million input tokens read).