HarnessEval-W: Agentifying the Evaluation of Visual Worlds

Summary (Overview)

• Novel Agentic Evaluation Paradigm: HarnessEval-W introduces an agentified evaluation pipeline that brings the "harness" paradigm from LLM evaluation to world model benchmarking, replacing static rubrics with a hierarchical agentic workflow that produces transparent, verifiable reasoning traces for every evaluation.

• Comprehensive Benchmark: The benchmark comprises 330 evaluation cases organized around three core axes of world model capability—Observation Quality, Transition Correctness, and World Persistence—derived from a formal factorization of world model prediction.

• Superior Human Alignment: Evaluations from HarnessEval-W achieve strong alignment with human preferences (Spearman rank correlation ρ = 0.93 for Intentional Transition, ρ = 0.87 for Physical Transition), substantially outperforming existing benchmarks like WBench.

• 18 World Models Evaluated: The benchmark evaluates a diverse set of 18 representative world models spanning text-driven video generators, camera-pose-conditioned models, and native action-controlled simulators, with Seedance 2.0 ranking first overall (75.5/100).

• Living Benchmark: The pipeline is open-sourced as an executable, extensible system designed for recursive self-improvement—capable of growing new skills and evaluation cases as world models evolve.

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses a critical fragility in world model evaluation: while humans can naturally spot violations of physics, causality, and world-state coherence in generated videos, no existing benchmark automates this capability. Traditional evaluation metrics compute scores "brute-force," producing scalar outputs without any reasoning chain that can be examined or verified.

Theoretical Framework

The authors formalize interactive world models through a probabilistic factorization. Given initial observations $\{o_i\}_{i=-T}^{0}$ and future actions $\{a_i\}_{i=0}^{t-1}$, the distribution of future observations $\{o_i\}_{i=1}^{t}$ is decomposed as:

$$P(o_{1}, \dots, o_{t} \mid o_{-T}, \dots, o_{0}; a_{0}, \dots, a_{t-1}) \propto P(s_{0} \mid o_{-T}, \dots, o_{0}) \prod_{i=1}^{t} S(o_{i} \mid s_{i}) T(s_{i} \mid s_{i-1}, a_{i-1}), \tag{1}$$

Where:

• $S(o_i \mid s_i)$ models observation likelihood conditioned on current state
• $T(s_i \mid s_{i-1}, a_{i-1})$ represents state transition conditioned on the given action
• $P(s_0 \mid o_{-T}, \dots, o_0)$ is the initial state distribution inferred from historical observations

This factorization exposes three fundamental capabilities evaluated in HarnessEval-W:

1. Observation Quality: Rendering observations from current state (S)
2. Transition Correctness: Updating state under actions (T)
3. World Persistence: Maintaining coherent state sequence over time ($\{s_i\}_{i=0}^{t}$)

Methodology

Hierarchical Agentic Evaluation

The evaluation pipeline operates hierarchically, decomposing monolithic evaluation into actionable subproblems:

1. Case-specific Skill Routing: The system interprets case context (initial image, action prompt, evaluation setting) and routes to appropriate skills from a predefined skill library. This routing is model-independent, ensuring fairness across all evaluated models.

2. Sub-agent Reasoning: Each high-level skill decomposes into measurable sub-questions answered by specialized sub-agents. For example, the Intentional Change Verifier decomposes into eight sub-questions:

• Target visibility verification
• Visible transition occurrence
• Target correctness
• Final state validity
• Anchor preservation
• Extra event detection
• Overall judgeability

3. Evidence Aggregation: Parent agent validates evidence from all sub-agents and merges results into a structured evidence tree with a final verdict.

Evaluation Axes and Settings

The benchmark tests eight detailed settings across three evaluation axes:

Case Construction Pipeline

The benchmark construction uses an agentic pipeline with:

1. Scene Taxonomy Sampling: Six complementary axes (Environment, Foreground, Midground, Scene Density, Appearance, Perspective)
2. Probe Family Assignment: Six probe families corresponding to evaluation settings
3. Agentic Case Authoring: Image generator, image-grounded planner, and case validator agents
4. Benchmark Composition: 330 cases spanning diverse environments, entities, and interaction patterns

Empirical Validation / Results

Main Leaderboard

All 18 models evaluated on identical 330 cases, grouped by conditioning interface:

Human Alignment

The benchmark demonstrates strong alignment with human preferences:

• Intentional Transition: Spearman rank correlation ρ = 0.93 (Kendall τ = 0.82)
• Physical Transition: ρ = 0.87 (τ = 0.74)

Comparison with WBench

When compared to WBench protocols on identical videos:

Robustness Analysis

HarnessEval-W shows consistent evaluation across repeated runs:

• Slopes of fitted curves: 9.6–10.8 (vs. WBench's 11.2–21.0)
• Correlations with human strength: 0.928–0.964 (vs. WBench's 0.646–0.780)
• Envelope span: 0.33 Bradley–Terry units (vs. WBench's 1.61 units, 4.9× wider)

Correlation Analysis

Key correlations across evaluation axes:

• Render Quality vs. Physical Observation: r = −0.04 (nearly uncorrelated)
• Intentional and Physical Transition: r = 0.98 (strongly coupled)
• Exploratory Transition vs. Intentional: r = −0.15
• Exploratory Transition vs. Physical: r = −0.18

Fine-tuning Analysis

When text-to-video models are fine-tuned into action-conditioned models:

• DreamX-World (from Wan 2.2): +4.8 Exploratory, +7.8 Revisit, +3.5 Offscreen, but −11.9 Intentional, −7.2 Physical
• HY-WorldPlay (from Hunyuan Video 1.5): +8.4 Revisit, but −24.2 Intentional, −11.2 Physical

This suggests fine-tuning emphasizes exploration-style trajectories over commanded interventions.

Theoretical and Practical Implications

Theoretical Contributions

1. Formalization of World Model Evaluation: Equation (1) provides a unified mathematical framework for understanding world model capabilities, decomposing the prediction problem into observation, transition, and persistence components.

2. Agentic Evaluation Paradigm: The paper establishes the concept of agentified evaluation, where evaluation is treated as a hierarchical workflow with decomposable subproblems rather than a fixed rubric.

3. Interpretability: The evidence tree structure makes evaluation auditable, allowing researchers to trace every score back to specific sub-questions and visual evidence.

Practical Implications

1. Actionable Diagnostics: The benchmark provides fine-grained identification of exact failure modes, guiding model development.

2. Cross-Interface Comparison: By evaluating models with different interfaces on identical cases, the benchmark enables fair comparison across conditioning types.

3. Living Benchmark: The extensible skill library and self-improving evaluation loop address the evolving nature of world models.

Conclusion

HarnessEval-W represents a paradigm shift in world model evaluation, moving from black-box scalar metrics to transparent, reasoning-based evaluation. The benchmark demonstrates that:

1. Agentic evaluation can align closely with human judgment while providing interpretable, verifiable results
2. Adaptive testing reveals capability distributions that static benchmarks miss
3. The evaluation harness can evolve with the models it measures, addressing the challenge of benchmark obsolescence

The authors propose three future directions:

1. Test-time scaling for agentic benchmarks, applying more compute to deeper analysis
2. Scaling skill libraries to cover increasingly complex world scenarios
3. Recursively self-improving agentic benchmarks that acquire new skills autonomously

The benchmark is open-sourced as a living system, inviting community contribution to grow new skills and evaluation cases as world models evolve.