Summary (Overview)

• InteractiveWorld is the first verifiable, human-centric closed-loop interaction environment for evaluating egocentric AI assistants, built on NVIDIA Isaac Sim with over 8,000 objects and 50 scenes.
• InteractiveBench is the first dynamic, multi-turn benchmark for egocentric AI assistants, comprising 120 human-reviewed tasks across six categories: Guidance, Planning, Accessibility, Decision, Prevention, and Finding.
• The framework simulates realistic human–assistant collaboration via a three-layer architecture: Physical Layer (realistic humans and physics), Human Action Layer (autonomous navigation and manipulation), and Semantic Layer (natural language grounding into 36 action primitives).
• Experiments with 12 frontier MLLMs reveal that even the strongest model (Claude Fable 5.1) achieves only 41.7% average task success, with particularly poor performance on long-horizon Guidance (20.0%) and Planning (25.0%) tasks.
• Key bottlenecks identified: perception and memory errors account for 70.4%–85.7% of first errors, and communication quality critically bounds overall task success.

Introduction and Theoretical Foundation

The paper addresses a fundamental question: Can today's MLLMs truly help people in the physical world? Egocentric AI assistants on wearable devices share the user's first-person view and provide natural-language guidance, avoiding physical control—the human acts while the assistant focuses on perception, memory, reasoning, and communication.

Limitations of existing benchmarks:

• Real-world experiments are costly and hard to scale
• Existing benchmarks are almost entirely static: models watch pre-recorded first-person video and answer questions, with no feedback loop
• Static benchmarks cannot test two critical abilities:

(1) Closed-loop adaptation: The assistant must continually adapt its guidance to the evolving scene and human responses, since its instructions directly change the human's actions and what it sees next. (2) Compositional capabilities: The assistant must combine skills such as tracking object states, remembering events, and communicating clearly, whereas static benchmarks typically test one skill at a time.

Three key challenges for human-centric simulation:

1. Human-centric egocentric view — the assistant observes through the human's first-person view, which shifts with body postures and hand movements
2. Reliable execution of everyday actions — humans can reliably execute actions (GRAB, OPEN) given high-level intents
3. Autonomous translation of guidance into actions — humans translate free-form guidance into executable actions based on scene observations

Methodology

Three-Layer Architecture

1. Physical Layer: Uses SMPL+H for full-body and hand modeling with a head-mounted camera for first-person vision. The human model is driven kinematically to prevent random falls. The system pre-checks navigation paths and arm reachability, automatically corrects minor collisions, and rejects unsafe actions with clear failure reasons.

2. Human Action Layer: Enables high-level actions through two stages:

• Collision-Aware Auto-Navigation: Identifies reachable standing positions, constructs obstacle maps, and plans safe paths using A* search (Hart et al., 1968)
• Target-Driven Adaptive Manipulation: Adapts posture (stand, lean, squat) and plans collision-free arm trajectories based on object shape, functional parts, and task

3. Semantic Layer: Defines 36 executable action primitives covering navigation, observation, manipulation, and daily activities. Commands specify only actions and targets (e.g., [Grab]<cup> or [PutInside]<pizza> <oven>) and are translated into goals for the Human Action Layer.

Evaluation Protocol

Each task evaluation consists of:

• Initial Scene: Complete starting world with scene layout, human position, and object states
• Warmup: Human performs predefined actions to establish context
• Closed-loop Interaction: Assistant uses egocentric view + history to choose between interrupt (guidance), silent, or done; Human issues action commands based on guidance
• Termination: When Assistant outputs $done$ or maximum interaction limit reached
• Scoring: Task success requires all predefined core states to be verified from actual environment states

Task Categories

Empirical Validation / Results

Overall Performance (Table 2)

Verified: Oracle Assistant achieves 100% pass rate; 0% without assistance.

Key Findings

Finding 1: Closed-loop assistance is far from solved — Even the strongest model helps complete fewer than half of tasks (41.7%), with all others at or below 33.3%.

Finding 2: Sustaining the loop is the core difficulty — Models make good one-shot decisions (85% on Decision) but break down on long-horizon tasks requiring continuous tracking and replanning (20% Guidance, 25% Planning). Performance drops quickly as task length increases.

Finding 3: Perception and memory are the primary bottlenecks — Visual perception and memory errors account for 70.4%–85.7% of first errors across all models. A single misperception cascades into all subsequent instructions.

Finding 4: Communication is critical — GLM-5.3-Flash and Claude Fable 5.1 have nearly identical average capability scores (58.2% vs. 59.2%), yet GLM scores markedly lower on communication (69.8% vs. 79.6%) and completes 11.7 points fewer tasks. Success is bounded by the weakest capability in the chain, not the average.

Finding 5: Action efficiency is severely lacking — Models require 2.35×–3.66× the Oracle's action cost on Accessibility and Guidance tasks. Efficiency is task-dependent: Claude Fable 5.1 is least efficient on Accessibility (6.75×) yet highly efficient on Prevention (1.20×).

Capability Assessment (Table 3 – Selected Models)

Ablation Studies

Memory Augmentation: Adding a hidden $Memory$ field improves Prevention and Finding by 5–15% across models, but effects are mixed elsewhere—Planning drops 5% for two models. Simple memory augmentation is insufficient for reliable long-horizon assistance.

Human Agent Impact: Cross-evaluating 16 Assistant–Human pairings shows performance varies with the Human partner, but rankings remain stable (Fable 5.1 consistently first). Gemini 3.1 Pro serves as a good Human proxy—all four Assistants achieve their highest MCR when paired with it.

Theoretical and Practical Implications

For evaluation methodology: InteractiveBench represents a paradigm shift from static video QA to closed-loop, state-verifiable evaluation. It captures abilities that static benchmarks cannot measure: sustained attention, dynamic replanning, proactive information-seeking, and communication effectiveness over many turns.

For model development: The results reveal that current MLLMs' foundational visual capabilities are insufficient for real-world interaction. The finding that perception and memory errors dominate (70.4%–85.7% of failures) suggests research should prioritize:

• Robust visual grounding in dynamic, egocentric scenes
• Native memory mechanisms for long-horizon state tracking
• Communication training that translates plans into executable, unambiguous instructions

For practical deployment: The emergent proactive behaviors (asking users to look around, flip objects to reveal labels, request spatial information) demonstrate that models can actively shape their observations—an essential capability for real-world assistance where key information is often occluded.

Conclusion

The paper introduces InteractiveWorld, the first verifiable human-centric interaction environment, and InteractiveBench, a 120-task dynamic benchmark for egocentric AI assistants. The three-layer architecture (Physical, Human Action, Semantic) realistically simulates human–assistant collaboration with state-verifiable rewards.

The evaluation of 12 frontier MLLMs reveals substantial gaps: the strongest model succeeds on only 41.7% of tasks, with perception, memory, and communication identified as key bottlenecks. The work provides a verifiable platform and rigorous evaluation set, laying a foundation for future research on interactive MLLM assistants.

Future directions include:

• Training native memory mechanisms for dynamic interaction
• Behavioral calibration to better align simulated Human agents with real users
• Optimizing interaction efficiency alongside task success rates