# InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

> InteractiveWorld and InteractiveBench, the first verifiable closed-loop egocentric AI assistant benchmark, show even the strongest MLLM succeeds on only 41.7% of tasks, with perception and memory errors causing most failures.

- **Source:** [arXiv](https://arxiv.org/abs/2610.33966)
- **Published:** 2026-10-28
- **Permalink:** https://picx.dev/p/Sq3bML
- **Whiteboard:** https://picx.dev/p/Sq3bML/image

## Summary

## Summary (Overview)

- **InteractiveWorld** is the first verifiable, human-centric closed-loop interaction environment for evaluating egocentric AI assistants, built on NVIDIA Isaac Sim with over 8,000 objects and 50 scenes.
- **InteractiveBench** is the first dynamic, multi-turn benchmark for egocentric AI assistants, comprising 120 human-reviewed tasks across six categories: Guidance, Planning, Accessibility, Decision, Prevention, and Finding.
- The framework simulates realistic human–assistant collaboration via a three-layer architecture: Physical Layer (realistic humans and physics), Human Action Layer (autonomous navigation and manipulation), and Semantic Layer (natural language grounding into 36 action primitives).
- Experiments with 12 frontier MLLMs reveal that even the strongest model (Claude Fable 5.1) achieves only 41.7% average task success, with particularly poor performance on long-horizon Guidance (20.0%) and Planning (25.0%) tasks.
- Key bottlenecks identified: perception and memory errors account for 70.4%–85.7% of first errors, and communication quality critically bounds overall task success.

## Introduction and Theoretical Foundation

The paper addresses a fundamental question: **Can today's MLLMs truly help people in the physical world?** Egocentric AI assistants on wearable devices share the user's first-person view and provide natural-language guidance, avoiding physical control—the human acts while the assistant focuses on perception, memory, reasoning, and communication.

**Limitations of existing benchmarks:**
- Real-world experiments are costly and hard to scale
- Existing benchmarks are almost entirely static: models watch pre-recorded first-person video and answer questions, with no feedback loop
- Static benchmarks cannot test two critical abilities:

> (1) **Closed-loop adaptation**: The assistant must continually adapt its guidance to the evolving scene and human responses, since its instructions directly change the human's actions and what it sees next. (2) **Compositional capabilities**: The assistant must combine skills such as tracking object states, remembering events, and communicating clearly, whereas static benchmarks typically test one skill at a time.

**Three key challenges for human-centric simulation:**
1. **Human-centric egocentric view** — the assistant observes through the human's first-person view, which shifts with body postures and hand movements
2. **Reliable execution of everyday actions** — humans can reliably execute actions (GRAB, OPEN) given high-level intents
3. **Autonomous translation of guidance into actions** — humans translate free-form guidance into executable actions based on scene observations

## Methodology

### Three-Layer Architecture

**1. Physical Layer:** Uses SMPL+H for full-body and hand modeling with a head-mounted camera for first-person vision. The human model is driven kinematically to prevent random falls. The system pre-checks navigation paths and arm reachability, automatically corrects minor collisions, and rejects unsafe actions with clear failure reasons.

**2. Human Action Layer:** Enables high-level actions through two stages:
- **Collision-Aware Auto-Navigation**: Identifies reachable standing positions, constructs obstacle maps, and plans safe paths using **A\* search** (Hart et al., 1968)
- **Target-Driven Adaptive Manipulation**: Adapts posture (stand, lean, squat) and plans collision-free arm trajectories based on object shape, functional parts, and task

**3. Semantic Layer:** Defines **36 executable action primitives** covering navigation, observation, manipulation, and daily activities. Commands specify only actions and targets (e.g., `[Grab]<cup>` or `[PutInside]<pizza> <oven>`) and are translated into goals for the Human Action Layer.

### Evaluation Protocol

Each task evaluation consists of:
- **Initial Scene**: Complete starting world with scene layout, human position, and object states
- **Warmup**: Human performs predefined actions to establish context
- **Closed-loop Interaction**: Assistant uses egocentric view + history to choose between interrupt (guidance), silent, or done; Human issues action commands based on guidance
- **Termination**: When Assistant outputs `$done$` or maximum interaction limit reached
- **Scoring**: Task success requires all predefined core states to be verified from actual environment states

### Task Categories

| Category | Evaluation focus | Example task |
|---|---|---|
| Guidance | Step-by-step guidance based on user's progress | Lab cleanup with 14 chemical bottles and broken glass |
| Planning | Steps within time/resource limits, flexible adjustment | Cooking for guests with allergies within 30 minutes |
| Accessibility | Scene description for blind/low-vision users | Kitchen cleanup for a blind user |
| Decision | Daily decisions using observations and knowledge | Choosing cheaper supplier quote for 40 filters |
| Prevention | Warnings before mistakes, avoiding unnecessary interruptions | Reminding user about open fridge/window before leaving |
| Finding | Finding missing objects via memory and guided search | Locating stored squeegees in a lidded box in another room |

## Empirical Validation / Results

### Overall Performance (Table 2)

| Model | AVG Pass (%) | Guidance Pass (%) | Planning Pass (%) | Decision Pass (%) | Prevention Pass (%) | Finding Pass (%) |
|---|---|---|---|---|---|---|
| **Claude Fable 5.1** | **41.7** | 20.0 | 25.0 | **85.0** | **50.0** | **40.0** |
| Claude Opus 5 | 33.3 | 15.0 | 25.0 | 70.0 | 30.0 | 35.0 |
| Muse Spark 1.1 | 31.7 | 20.0 | 25.0 | 55.0 | 30.0 | 25.0 |
| GPT-5.6 Sol | 30.8 | 20.0 | 25.0 | 70.0 | 15.0 | 25.0 |
| GPT-5.5 | 30.0 | 10.0 | 15.0 | 65.0 | 25.0 | 20.0 |
| GLM 5.3 Flash | 30.0 | 15.0 | 20.0 | 50.0 | 20.0 | 25.0 |
| Claude Opus 4.8 | 28.3 | 0.0 | 20.0 | 50.0 | 35.0 | 35.0 |
| Kimi K3 | 24.2 | 20.0 | 15.0 | 35.0 | 20.0 | 15.0 |
| Qwen 3.8 Max | 22.5 | 5.0 | 20.0 | 35.0 | 20.0 | 20.0 |
| Gemini 3.1 Pro | 20.8 | 10.0 | 10.0 | 50.0 | 10.0 | 10.0 |
| GPT-5.6 Luna | 20.0 | 5.0 | 5.0 | 40.0 | 30.0 | 20.0 |
| DeepSeek V4.1 Flash | 12.5 | 0.0 | 5.0 | 35.0 | 0.0 | 10.0 |

*Verified: Oracle Assistant achieves 100% pass rate; 0% without assistance.*

### Key Findings

**Finding 1: Closed-loop assistance is far from solved** — Even the strongest model helps complete fewer than half of tasks (41.7%), with all others at or below 33.3%.

**Finding 2: Sustaining the loop is the core difficulty** — Models make good one-shot decisions (85% on Decision) but break down on long-horizon tasks requiring continuous tracking and replanning (20% Guidance, 25% Planning). Performance drops quickly as task length increases.

**Finding 3: Perception and memory are the primary bottlenecks** — Visual perception and memory errors account for **70.4%–85.7% of first errors** across all models. A single misperception cascades into all subsequent instructions.

**Finding 4: Communication is critical** — GLM-5.3-Flash and Claude Fable 5.1 have nearly identical average capability scores (58.2% vs. 59.2%), yet GLM scores markedly lower on communication (69.8% vs. 79.6%) and completes 11.7 points fewer tasks. Success is bounded by the **weakest capability in the chain**, not the average.

**Finding 5: Action efficiency is severely lacking** — Models require **2.35×–3.66×** the Oracle's action cost on Accessibility and Guidance tasks. Efficiency is task-dependent: Claude Fable 5.1 is least efficient on Accessibility (6.75×) yet highly efficient on Prevention (1.20×).

### Capability Assessment (Table 3 – Selected Models)

| Model | Objects | Spatial | OCR | Memory | Plan | Knowledge | Detect | Proactivity | Comm. | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Fable 5.1 | 63.0 | 61.2 | 62.5 | 58.9 | 56.0 | 54.9 | 51.9 | 44.4 | **79.6** | **59.2** |
| Claude Opus 5 | 54.9 | 57.1 | 59.8 | 46.4 | 55.5 | 54.4 | 54.9 | **53.4** | 80.1 | 57.4 |
| GLM 5.3 Flash | 59.3 | 56.7 | **69.3** | 49.9 | 54.6 | **61.3** | **56.8** | 45.9 | 69.8 | 58.2 |
| DeepSeek V4.1 Flash | 57.4 | 46.0 | 51.1 | 31.1 | 35.7 | 40.9 | 33.3 | 21.9 | 65.9 | 42.6 |

### Ablation Studies

**Memory Augmentation:** Adding a hidden `$Memory$` field improves Prevention and Finding by 5–15% across models, but effects are mixed elsewhere—Planning drops 5% for two models. Simple memory augmentation is insufficient for reliable long-horizon assistance.

**Human Agent Impact:** Cross-evaluating 16 Assistant–Human pairings shows performance varies with the Human partner, but rankings remain stable (Fable 5.1 consistently first). Gemini 3.1 Pro serves as a good Human proxy—all four Assistants achieve their highest MCR when paired with it.

## Theoretical and Practical Implications

**For evaluation methodology:** InteractiveBench represents a paradigm shift from static video QA to closed-loop, state-verifiable evaluation. It captures abilities that static benchmarks cannot measure: sustained attention, dynamic replanning, proactive information-seeking, and communication effectiveness over many turns.

**For model development:** The results reveal that current MLLMs' foundational visual capabilities are insufficient for real-world interaction. The finding that perception and memory errors dominate (70.4%–85.7% of failures) suggests research should prioritize:
- Robust visual grounding in dynamic, egocentric scenes
- Native memory mechanisms for long-horizon state tracking
- Communication training that translates plans into executable, unambiguous instructions

**For practical deployment:** The emergent proactive behaviors (asking users to look around, flip objects to reveal labels, request spatial information) demonstrate that models can actively shape their observations—an essential capability for real-world assistance where key information is often occluded.

## Conclusion

The paper introduces **InteractiveWorld**, the first verifiable human-centric interaction environment, and **InteractiveBench**, a 120-task dynamic benchmark for egocentric AI assistants. The three-layer architecture (Physical, Human Action, Semantic) realistically simulates human–assistant collaboration with state-verifiable rewards.

The evaluation of 12 frontier MLLMs reveals substantial gaps: the strongest model succeeds on only 41.7% of tasks, with perception, memory, and communication identified as key bottlenecks. The work provides a verifiable platform and rigorous evaluation set, laying a foundation for future research on interactive MLLM assistants.

**Future directions** include:
- Training native memory mechanisms for dynamic interaction
- Behavioral calibration to better align simulated Human agents with real users
- Optimizing interaction efficiency alongside task success rates

---

_Markdown view of https://picx.dev/p/Sq3bML, served by PicX — AI-generated visual whiteboard summaries of research papers._