# Show-Harness: Just a VLM Agent Can Play Robots

> Show-Harness shows that a compact, discrete semantic action interface—not model scale—unlocks strong embodied control from foundation VLMs, beating VLA baselines across tasks, environments, and embodiments.

- **Source:** [arXiv](https://arxiv.org/abs/2609.10522)
- **Published:** 2026-09-11
- **Permalink:** https://picx.dev/p/DLZIHU
- **Whiteboard:** https://picx.dev/p/DLZIHU/image

## Summary

## Summary (Overview)

- **Show-Harness** is a model-agnostic "Embodied Harness" that enables foundation Vision-Language Models (VLMs) to control robots through a compact, discrete semantic action interface, rather than regressing continuous motor commands or relying on opaque downstream controllers.
- It demonstrates two operational modes: **zero-shot control** with closed-source frontier VLMs (e.g., Gemini-3.1 Pro) and **efficient fine-tuning** of small open-source VLMs (e.g., Qwen3.5-2B) with just a few GPU-hours.
- The system shows strong generalization across tasks, environments, and embodiments, outperforming representative VLA (Vision-Language-Action) and agentic baselines in real-robot experiments.
- It introduces **GUMI** (GUI Manipulation Interface), which allows humans and agents to collect robot demonstrations through a graphical interface using the same semantic action space, eliminating the need for specialized teleoperation hardware.
- Key findings suggest that the *interface*—not model capacity or embodiment-specific pretraining—is the critical factor in unlocking embodied capability from foundation VLMs.

## Introduction and Theoretical Foundation

### Background and Motivation
- Foundation VLMs encode substantial knowledge about the physical world (object recognition, spatial relations, long-horizon goals) but translating this into robot control remains challenging.
- **VLA models** (e.g., π₀.₅, GR00T) fine-tune VLMs to regress continuous actions, but this collapses broad semantic knowledge into opaque sensorimotor mappings requiring repeated adaptation.
- **Hierarchical/programmatic systems** let VLMs produce high-level abstractions (subtasks, API calls), but physical control is mediated by downstream controllers, weakening the link between semantic intent and execution.

### Core Thesis
The authors argue that bringing foundation-model intelligence into the physical world requires a *suitable interface*: an action space that is:
1. **Semantically meaningful** to the VLM (interpretable symbols, not numeric targets)
2. **Sufficiently fine-grained** for direct physical control (each action produces a small, observable change)

### Theoretical Foundation
The approach is grounded in the observation that frontier VLMs perform very differently under different levels of action abstraction. Human-designed abstractions like `place(object, target)` improve reliability, but composing low-level APIs remains difficult. Show-Harness exposes the "how" itself through fine-grained semantic units whose physical realization is deterministic and transparent, keeping the VLM directly responsible for physical decisions.

## Methodology

### Semantic Action Space
The shared action space $\mathcal{A}$ consists of discrete semantic units:
- **MV_FWD/MV_BACK, MV_LEFT/MV_RIGHT, MV_UP/MV_DOWN**: translate the end-effector one step in the specified direction relative to a reference view
- **ROTATE_CW/ROTATE_CCW** (with axis x, y, or z): incrementally rotate the end-effector
- **GRASP/RELEASE**: close/open the gripper
- **DONE**: indicate task completion

The vocabulary is designed to be:
1. **Incremental**: Each unit induces a small, localized physical change, keeping the VLM situated in the control loop
2. **Interpretable and embodiment-agnostic**: Semantic symbols, not numeric targets
3. **Visually grounded**: Directions defined relative to observable views

### Embodiment Grounding
Each semantic action $a_t$ is deterministically grounded by an embodiment-specific interpreter $g_E$:

$$s_{t+1} = \Pi_E(\mathbf{x}_t + \sigma_t R_E d_a, \exp(\theta_t [R_E r_a]_\times) Q_t) \tag{4}$$

Where:
- $\mathbf{x}_t \in \mathbb{R}^3$ and $Q_t \in SO(3)$ denote position and orientation
- $d_a$ and $r_a$ encode translation and rotation (with $r_a \in \{\pm e_x, \pm e_y, \pm e_z\}$ for rotation)
- $[\cdot]_\times$ is the skew-symmetric matrix operator
- $\sigma_t$ and $\theta_t$ are calibrated translation/rotation increments
- $R_E$ maps semantic directions to the motion frame of embodiment $E$
- $\Pi_E$ enforces workspace and per-step limits

### Harness Architecture
The harness organizes interaction into three configurable stages with plugins:

| Stage | Plugin | Function |
|---|---|---|
| **Perception** | Multi-View Guidance | Guides VLM on camera view roles |
| | Proprioception | Translates robot state into textual feedback |
| **Reasoning** | Subtask Planning | Maintains ordered subtask plan |
| | Situated Planning | Defers uncertain decisions, triggers replanning |
| | Action Chunking | Adaptively chunks actions for efficiency |
| | Adaptive Step | Adjusts step size (2cm fine / 4cm coarse) |
| | Visual Prompt | Highlights visual targets via a dedicated model call |
| **Action** | Action History | Summarizes recent actions, discourages oscillation |
| | Failure Recovery | Detects grasp failures and triggers recovery |

### Two Modes
1. **Zero-shot (ZS) mode**: Frontier VLM (Gemini-3.1 Pro) directly controls robots without fine-tuning
2. **Fine-tuning (FT) mode**: Small VLM (Qwen3.5-2B) trained with rank-64 LoRA on ~3% of parameters, minimizing:

$$\mathcal{L}(\theta) = -\sum_{(\ell, o, h, a) \in \mathcal{D}} \log \pi_\theta(a \mid \Phi_{\mathcal{P}_{\min}}(\ell, o, h)) \tag{5}$$

### GUMI (GUI Manipulation Interface)
- Exposes the same semantic action units as labeled controls/keystrokes
- Enables humans, computer-use agents, and VLM agents to collect demonstrations
- Records policy-ready pairs $(o_t, a_t)$ and supports cross-embodiment reuse

## Empirical Validation / Results

### Experimental Setup
- **Hardware**: 7-DoF Franka Research 3 arm (exocentric + wrist cameras), bimanual AgileX rig (6-DoF arms, 3 cameras)
- **Tasks**: 10 real-robot manipulation tasks pairing 5 objects (block, banana, tennis ball, teddy bear, chess piece) with 2 receptacles (plate, bowl)
- **Data**: 164 real-robot episodes (7.8K steps) + 230 simulated episodes (13.5K steps)

### Main Results (Table 2)

| Setting | VLA (π₀.₅) | VLA (GR00T) | H-VLA | G-VLA | CaP-X | RATS | **ZS** | **FT** |
|---|---|---|---|---|---|---|---|---|
| **Cross-Task Avg (%)** | 39.0 | 35.0 | 50.0 | 13.0 | 44.0 | 57.0 | **89.0** | **86.0** |
| **Cross-Environment Avg (%)** | 40.0 | 34.0 | 63.8 | 15.0 | 52.5 | 65.0 | **100.0** | **88.0** |
| **Cross-Embodiment Avg (%)** | 41.0 | 36.0 | 49.0 | 11.0 | 43.0 | 52.0 | **93.0** | **87.0** |

Key observations:
- **Sim-to-real**: FT succeeds (13/20) using only simulated demonstrations; trainable VLA baselines fail completely (0/20)
- **Held-out objects** (teddy bear, chess piece): Show-Harness maintains high success (8-10/10) vs. VLA baselines (0-5/10)

### Capability Analysis

**Physical Adaptability:**
- **Fine-grained control**: Reducing interpreter step size from 2cm to 1cm improves ZS from 60%→82% and FT from 40%→65% (vs. π₀.₅ at 18%, reaching 62% only with additional training)
- **Rotation extrapolation**: FT reaches 70% at unseen 90° orientation (trained only on 0° and 45°); π₀.₅ reaches only 20%
- **Workspace shift**: Show-Harness degrades mildly; π₀.₅ drops sharply
- **Multi-arm coordination**: Joint action prediction improves success and eliminates collisions

**Semantic Adaptability:**
- **Reasoning tasks** (hidden object, arranging "SHOW"): ZS with Situated Planning achieves 85%; FT alone 10%; with Gemini-generated subtask instructions, FT rises to 70%
- **In-context learning**: ZS follows demonstrated order in 20/20 trials from video demonstrations

### Ablation Findings
- **Multi-View Guidance**: Essential for small objects requiring precise alignment
- **Proprioception**: Critical when visual cues are ambiguous
- **Subtask Planning**: Without it, success drops to 60% (model drags objects without lifting)
- **Adaptive Step**: Balances precision (fine steps) and efficiency (coarse steps)
- **Action History**: Prevents oscillation between opposing actions
- **Failure Recovery**: Detects empty grasps; without it, success drops to 72%
- **Action-space representation**: Semantic names + written conventions (A) nearly match arbitrary symbols + conventions (C); arbitrary symbols without conventions (D) fail (1/20 episodes, only 23.3% correct mappings inferred)

## Theoretical and Practical Implications

### Theoretical Contributions
1. **Interface-centric design**: The paper provides empirical evidence that the *action interface* is the key bottleneck in VLM-based robot control—not model capacity or embodiment-specific pretraining. A compact, semantically meaningful, yet fine-grained action space can unlock substantial embodied capability from existing foundation models.

2. **Semantic vs. continuous control**: The results challenge the dominant VLA paradigm (continuous action regression) by showing that discrete semantic actions, deterministically grounded, achieve stronger generalization with far less training data and compute.

3. **Unified human-agent interface**: The same semantic action space is operable by humans (via GUI), frontier agents (zero-shot), and small fine-tuned models, enabling scalable demonstration collection and cross-embodiment reuse.

### Practical Implications
- **Zero-shot robot control**: Closed-source frontier VLMs can control robots without any fine-tuning, providing a scalable path that inherits advances in foundation models.
- **Low-cost deployment**: Small (2B-scale) open-source models can be adapted with just a few GPU-hours, making robot control accessible without large-scale compute.
- **Sim-to-real transfer**: The semantic interface enables training on simulated demonstrations that transfer to real robots, where continuous VLA baselines fail completely.
- **Safe and interpretable control**: Embodiment-specific interpreters enforce workspace limits and safety bounds, while the semantic action space remains transparent and auditable.

## Conclusion

Show-Harness demonstrates that the right interface can unlock substantial embodied capability from foundation VLMs without requiring additional model capacity or costly embodiment-specific pretraining. Key takeaways:

1. **Semantic action units** (incremental, interpretable, visually grounded) provide an effective bridge between VLM reasoning and physical robot control.
2. **Two complementary modes**—zero-shot frontier VLMs and fine-tuned small models—both outperform representative VLA and agentic baselines across task, environment, and embodiment shifts.
3. **GUMI** enables flexible, hardware-free demonstration collection, supporting human-agent collaboration and cross-embodiment data reuse.

### Limitations and Future Directions
- Currently evaluated primarily on single- and dual-arm manipulation with parallel-jaw grippers
- Future work: extending to humanoids or dexterous hands, adding tactile and force feedback modalities for contact-rich interaction, enriching perception with additional embodied sensors

---

_Markdown view of https://picx.dev/p/DLZIHU, served by PicX — AI-generated visual whiteboard summaries of research papers._