Summary (Overview)

• Show-Harness is a model-agnostic "Embodied Harness" that enables foundation Vision-Language Models (VLMs) to control robots through a compact, discrete semantic action interface, rather than regressing continuous motor commands or relying on opaque downstream controllers.
• It demonstrates two operational modes: zero-shot control with closed-source frontier VLMs (e.g., Gemini-3.1 Pro) and efficient fine-tuning of small open-source VLMs (e.g., Qwen3.5-2B) with just a few GPU-hours.
• The system shows strong generalization across tasks, environments, and embodiments, outperforming representative VLA (Vision-Language-Action) and agentic baselines in real-robot experiments.
• It introduces GUMI (GUI Manipulation Interface), which allows humans and agents to collect robot demonstrations through a graphical interface using the same semantic action space, eliminating the need for specialized teleoperation hardware.
• Key findings suggest that the interface—not model capacity or embodiment-specific pretraining—is the critical factor in unlocking embodied capability from foundation VLMs.

Introduction and Theoretical Foundation

Background and Motivation

• Foundation VLMs encode substantial knowledge about the physical world (object recognition, spatial relations, long-horizon goals) but translating this into robot control remains challenging.
• VLA models (e.g., π₀.₅, GR00T) fine-tune VLMs to regress continuous actions, but this collapses broad semantic knowledge into opaque sensorimotor mappings requiring repeated adaptation.
• Hierarchical/programmatic systems let VLMs produce high-level abstractions (subtasks, API calls), but physical control is mediated by downstream controllers, weakening the link between semantic intent and execution.

Core Thesis

The authors argue that bringing foundation-model intelligence into the physical world requires a suitable interface: an action space that is:

1. Semantically meaningful to the VLM (interpretable symbols, not numeric targets)
2. Sufficiently fine-grained for direct physical control (each action produces a small, observable change)

Theoretical Foundation

The approach is grounded in the observation that frontier VLMs perform very differently under different levels of action abstraction. Human-designed abstractions like place(object, target) improve reliability, but composing low-level APIs remains difficult. Show-Harness exposes the "how" itself through fine-grained semantic units whose physical realization is deterministic and transparent, keeping the VLM directly responsible for physical decisions.

Methodology

Semantic Action Space

The shared action space $\mathcal{A}$ consists of discrete semantic units:

• MV_FWD/MV_BACK, MV_LEFT/MV_RIGHT, MV_UP/MV_DOWN: translate the end-effector one step in the specified direction relative to a reference view
• ROTATE_CW/ROTATE_CCW (with axis x, y, or z): incrementally rotate the end-effector
• GRASP/RELEASE: close/open the gripper
• DONE: indicate task completion

The vocabulary is designed to be:

1. Incremental: Each unit induces a small, localized physical change, keeping the VLM situated in the control loop
2. Interpretable and embodiment-agnostic: Semantic symbols, not numeric targets
3. Visually grounded: Directions defined relative to observable views

Embodiment Grounding

Each semantic action $a_t$ is deterministically grounded by an embodiment-specific interpreter $g_E$:

$$s_{t+1} = \Pi_E(\mathbf{x}_t + \sigma_t R_E d_a, \exp(\theta_t [R_E r_a]_\times) Q_t) \tag{4}$$

Where:

• $\mathbf{x}_t \in \mathbb{R}^3$ and $Q_t \in SO(3)$ denote position and orientation
• $d_a$ and $r_a$ encode translation and rotation (with $r_a \in \{\pm e_x, \pm e_y, \pm e_z\}$ for rotation)
• $[\cdot]_\times$ is the skew-symmetric matrix operator
• $\sigma_t$ and $\theta_t$ are calibrated translation/rotation increments
• $R_E$ maps semantic directions to the motion frame of embodiment $E$
• $\Pi_E$ enforces workspace and per-step limits

Harness Architecture

The harness organizes interaction into three configurable stages with plugins:

Two Modes

1. Zero-shot (ZS) mode: Frontier VLM (Gemini-3.1 Pro) directly controls robots without fine-tuning
2. Fine-tuning (FT) mode: Small VLM (Qwen3.5-2B) trained with rank-64 LoRA on ~3% of parameters, minimizing:

$$\mathcal{L}(\theta) = -\sum_{(\ell, o, h, a) \in \mathcal{D}} \log \pi_\theta(a \mid \Phi_{\mathcal{P}_{\min}}(\ell, o, h)) \tag{5}$$

GUMI (GUI Manipulation Interface)

• Exposes the same semantic action units as labeled controls/keystrokes
• Enables humans, computer-use agents, and VLM agents to collect demonstrations
• Records policy-ready pairs $(o_t, a_t)$ and supports cross-embodiment reuse

Empirical Validation / Results

Experimental Setup

• Hardware: 7-DoF Franka Research 3 arm (exocentric + wrist cameras), bimanual AgileX rig (6-DoF arms, 3 cameras)
• Tasks: 10 real-robot manipulation tasks pairing 5 objects (block, banana, tennis ball, teddy bear, chess piece) with 2 receptacles (plate, bowl)
• Data: 164 real-robot episodes (7.8K steps) + 230 simulated episodes (13.5K steps)

Main Results (Table 2)

Key observations:

• Sim-to-real: FT succeeds (13/20) using only simulated demonstrations; trainable VLA baselines fail completely (0/20)
• Held-out objects (teddy bear, chess piece): Show-Harness maintains high success (8-10/10) vs. VLA baselines (0-5/10)

Capability Analysis

Physical Adaptability:

• Fine-grained control: Reducing interpreter step size from 2cm to 1cm improves ZS from 60%→82% and FT from 40%→65% (vs. π₀.₅ at 18%, reaching 62% only with additional training)
• Rotation extrapolation: FT reaches 70% at unseen 90° orientation (trained only on 0° and 45°); π₀.₅ reaches only 20%
• Workspace shift: Show-Harness degrades mildly; π₀.₅ drops sharply
• Multi-arm coordination: Joint action prediction improves success and eliminates collisions

Semantic Adaptability:

• Reasoning tasks (hidden object, arranging "SHOW"): ZS with Situated Planning achieves 85%; FT alone 10%; with Gemini-generated subtask instructions, FT rises to 70%
• In-context learning: ZS follows demonstrated order in 20/20 trials from video demonstrations

Ablation Findings

• Multi-View Guidance: Essential for small objects requiring precise alignment
• Proprioception: Critical when visual cues are ambiguous
• Subtask Planning: Without it, success drops to 60% (model drags objects without lifting)
• Adaptive Step: Balances precision (fine steps) and efficiency (coarse steps)
• Action History: Prevents oscillation between opposing actions
• Failure Recovery: Detects empty grasps; without it, success drops to 72%
• Action-space representation: Semantic names + written conventions (A) nearly match arbitrary symbols + conventions (C); arbitrary symbols without conventions (D) fail (1/20 episodes, only 23.3% correct mappings inferred)

Theoretical and Practical Implications

Theoretical Contributions

1. Interface-centric design: The paper provides empirical evidence that the action interface is the key bottleneck in VLM-based robot control—not model capacity or embodiment-specific pretraining. A compact, semantically meaningful, yet fine-grained action space can unlock substantial embodied capability from existing foundation models.

2. Semantic vs. continuous control: The results challenge the dominant VLA paradigm (continuous action regression) by showing that discrete semantic actions, deterministically grounded, achieve stronger generalization with far less training data and compute.

3. Unified human-agent interface: The same semantic action space is operable by humans (via GUI), frontier agents (zero-shot), and small fine-tuned models, enabling scalable demonstration collection and cross-embodiment reuse.

Practical Implications

• Zero-shot robot control: Closed-source frontier VLMs can control robots without any fine-tuning, providing a scalable path that inherits advances in foundation models.
• Low-cost deployment: Small (2B-scale) open-source models can be adapted with just a few GPU-hours, making robot control accessible without large-scale compute.
• Sim-to-real transfer: The semantic interface enables training on simulated demonstrations that transfer to real robots, where continuous VLA baselines fail completely.
• Safe and interpretable control: Embodiment-specific interpreters enforce workspace limits and safety bounds, while the semantic action space remains transparent and auditable.

Conclusion

Show-Harness demonstrates that the right interface can unlock substantial embodied capability from foundation VLMs without requiring additional model capacity or costly embodiment-specific pretraining. Key takeaways:

1. Semantic action units (incremental, interpretable, visually grounded) provide an effective bridge between VLM reasoning and physical robot control.
2. Two complementary modes—zero-shot frontier VLMs and fine-tuned small models—both outperform representative VLA and agentic baselines across task, environment, and embodiment shifts.
3. GUMI enables flexible, hardware-free demonstration collection, supporting human-agent collaboration and cross-embodiment data reuse.

Limitations and Future Directions

• Currently evaluated primarily on single- and dual-arm manipulation with parallel-jaw grippers
• Future work: extending to humanoids or dexterous hands, adding tactile and force feedback modalities for contact-rich interaction, enriching perception with additional embodied sensors