# HarnessEval-W: Agentifying the Evaluation of Visual Worlds

> HarnessEval-W replaces static world model benchmarks with a hierarchical agentic pipeline, achieving human-level alignment (ρ=0.93) and exposing failure modes hidden by existing methods.

- **Source:** [arXiv](https://arxiv.org/abs/2608.16859)
- **Published:** 2026-08-19
- **Permalink:** https://picx.dev/p/R5jC88
- **Whiteboard:** https://picx.dev/p/R5jC88/image

## Summary

# HarnessEval-W: Agentifying the Evaluation of Visual Worlds

## Summary (Overview)

- **Novel Agentic Evaluation Paradigm**: HarnessEval-W introduces an agentified evaluation pipeline that brings the "harness" paradigm from LLM evaluation to world model benchmarking, replacing static rubrics with a hierarchical agentic workflow that produces transparent, verifiable reasoning traces for every evaluation.

- **Comprehensive Benchmark**: The benchmark comprises 330 evaluation cases organized around three core axes of world model capability—Observation Quality, Transition Correctness, and World Persistence—derived from a formal factorization of world model prediction.

- **Superior Human Alignment**: Evaluations from HarnessEval-W achieve strong alignment with human preferences (Spearman rank correlation ρ = 0.93 for Intentional Transition, ρ = 0.87 for Physical Transition), substantially outperforming existing benchmarks like WBench.

- **18 World Models Evaluated**: The benchmark evaluates a diverse set of 18 representative world models spanning text-driven video generators, camera-pose-conditioned models, and native action-controlled simulators, with Seedance 2.0 ranking first overall (75.5/100).

- **Living Benchmark**: The pipeline is open-sourced as an executable, extensible system designed for recursive self-improvement—capable of growing new skills and evaluation cases as world models evolve.

## Introduction and Theoretical Foundation

### Background and Motivation

The paper addresses a critical fragility in world model evaluation: while humans can naturally spot violations of physics, causality, and world-state coherence in generated videos, no existing benchmark automates this capability. Traditional evaluation metrics compute scores "brute-force," producing scalar outputs without any reasoning chain that can be examined or verified.

### Theoretical Framework

The authors formalize interactive world models through a probabilistic factorization. Given initial observations $\{o_i\}_{i=-T}^{0}$ and future actions $\{a_i\}_{i=0}^{t-1}$, the distribution of future observations $\{o_i\}_{i=1}^{t}$ is decomposed as:

$$
P(o_{1}, \dots, o_{t} \mid o_{-T}, \dots, o_{0}; a_{0}, \dots, a_{t-1}) \propto P(s_{0} \mid o_{-T}, \dots, o_{0}) \prod_{i=1}^{t} S(o_{i} \mid s_{i}) T(s_{i} \mid s_{i-1}, a_{i-1}), \tag{1}
$$

Where:
- $S(o_i \mid s_i)$ models observation likelihood conditioned on current state
- $T(s_i \mid s_{i-1}, a_{i-1})$ represents state transition conditioned on the given action
- $P(s_0 \mid o_{-T}, \dots, o_0)$ is the initial state distribution inferred from historical observations

This factorization exposes three fundamental capabilities evaluated in HarnessEval-W:
1. **Observation Quality**: Rendering observations from current state (S)
2. **Transition Correctness**: Updating state under actions (T)
3. **World Persistence**: Maintaining coherent state sequence over time ($\{s_i\}_{i=0}^{t}$)

## Methodology

### Hierarchical Agentic Evaluation

The evaluation pipeline operates hierarchically, decomposing monolithic evaluation into actionable subproblems:

**1. Case-specific Skill Routing**: The system interprets case context (initial image, action prompt, evaluation setting) and routes to appropriate skills from a predefined skill library. This routing is model-independent, ensuring fairness across all evaluated models.

**2. Sub-agent Reasoning**: Each high-level skill decomposes into measurable sub-questions answered by specialized sub-agents. For example, the Intentional Change Verifier decomposes into eight sub-questions:
- Target visibility verification
- Visible transition occurrence
- Target correctness
- Final state validity
- Anchor preservation
- Extra event detection
- Overall judgeability

**3. Evidence Aggregation**: Parent agent validates evidence from all sub-agents and merges results into a structured evidence tree with a final verdict.

### Evaluation Axes and Settings

The benchmark tests eight detailed settings across three evaluation axes:

| Evaluation Axis | Detail Evaluation Settings | Core World-state Question |
|---|---|---|
| Observation Quality | Render Quality | Is the rollout video coherent, stable, and sufficiently readable to serve as evidence? |
| Observation Quality | Physical Observation | Is each rendered frame from hidden state structurally and physically plausible? |
| Transition Correctness | Exploratory Transition | Does the requested viewpoint change occur while the world remains compatible? |
| Transition Correctness | Intentional Transition | Does the specified target change while protected state remains stable? |
| Transition Correctness | Physical Transition | Does a physical intervention produce the corresponding dynamical response? |
| World Persistence | Drift Resistance | Do invariant objects survive a long rollout? |
| World Persistence | Revisit Consistency | Does a place or entity remain compatible after leaving and returning? |
| World Persistence | Offscreen Evolution | Does an unobserved endogenous process continue rather than freeze or reset? |

### Case Construction Pipeline

The benchmark construction uses an agentic pipeline with:
1. **Scene Taxonomy Sampling**: Six complementary axes (Environment, Foreground, Midground, Scene Density, Appearance, Perspective)
2. **Probe Family Assignment**: Six probe families corresponding to evaluation settings
3. **Agentic Case Authoring**: Image generator, image-grounded planner, and case validator agents
4. **Benchmark Composition**: 330 cases spanning diverse environments, entities, and interaction patterns

## Empirical Validation / Results

### Main Leaderboard

All 18 models evaluated on identical 330 cases, grouped by conditioning interface:

| Model | Interface | Obs-Q | Obs-P | Trans-E | Trans-I | Trans-P | Pers-D | Pers-R | Pers-O | Overall |
|---|---|---|---|---|---|---|---|---|---|---|
| Seedance 2.0 | Prompt I2V | 83.6 | 61.8 | 80.2 | 81.8 | 63.5 | 79.8 | 76.8 | 68.9 | **75.5** |
| Wan 2.7 | Prompt I2V | 80.9 | 58.8 | 78.7 | 83.6 | 71.1 | 74.3 | 68.5 | 65.9 | 75.0 |
| Kling 3.0 | Prompt I2V | 82.1 | 60.6 | 79.1 | 82.6 | 63.2 | 77.2 | 75.4 | 66.2 | 74.4 |
| MiniMax H3 | Prompt I2V | 81.5 | 61.3 | 77.7 | 81.7 | 66.9 | 77.0 | 72.3 | 67.2 | 74.3 |
| Grok Imagine 1.5 | Prompt I2V | 85.1 | 60.6 | 76.4 | 80.2 | 66.7 | 79.1 | 70.9 | 64.5 | 73.4 |
| FLUX 3 | Prompt I2V | 81.7 | 61.6 | 76.4 | 79.0 | 63.2 | 77.1 | 70.2 | 67.4 | 72.2 |

### Human Alignment

The benchmark demonstrates strong alignment with human preferences:
- **Intentional Transition**: Spearman rank correlation ρ = 0.93 (Kendall τ = 0.82)
- **Physical Transition**: ρ = 0.87 (τ = 0.74)

### Comparison with WBench

When compared to WBench protocols on identical videos:

| Metric | HarnessEval-W | WBench |
|---|---|---|
| **Physical Pairwise Accuracy** | 71.7% | 31.9% |
| **Physical Draw Rate** | 1.8% | 52.2% |
| **Intentional Pairwise Accuracy** | 77.8% | 60.2% |
| **Intentional Draw Rate** | 11.1% | 36.1% |

### Robustness Analysis

HarnessEval-W shows consistent evaluation across repeated runs:
- Slopes of fitted curves: 9.6–10.8 (vs. WBench's 11.2–21.0)
- Correlations with human strength: 0.928–0.964 (vs. WBench's 0.646–0.780)
- Envelope span: 0.33 Bradley–Terry units (vs. WBench's 1.61 units, 4.9× wider)

### Correlation Analysis

Key correlations across evaluation axes:
- Render Quality vs. Physical Observation: r = −0.04 (nearly uncorrelated)
- Intentional and Physical Transition: r = 0.98 (strongly coupled)
- Exploratory Transition vs. Intentional: r = −0.15
- Exploratory Transition vs. Physical: r = −0.18

### Fine-tuning Analysis

When text-to-video models are fine-tuned into action-conditioned models:
- DreamX-World (from Wan 2.2): +4.8 Exploratory, +7.8 Revisit, +3.5 Offscreen, but −11.9 Intentional, −7.2 Physical
- HY-WorldPlay (from Hunyuan Video 1.5): +8.4 Revisit, but −24.2 Intentional, −11.2 Physical

This suggests fine-tuning emphasizes exploration-style trajectories over commanded interventions.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Formalization of World Model Evaluation**: Equation (1) provides a unified mathematical framework for understanding world model capabilities, decomposing the prediction problem into observation, transition, and persistence components.

2. **Agentic Evaluation Paradigm**: The paper establishes the concept of agentified evaluation, where evaluation is treated as a hierarchical workflow with decomposable subproblems rather than a fixed rubric.

3. **Interpretability**: The evidence tree structure makes evaluation auditable, allowing researchers to trace every score back to specific sub-questions and visual evidence.

### Practical Implications

1. **Actionable Diagnostics**: The benchmark provides fine-grained identification of exact failure modes, guiding model development.

2. **Cross-Interface Comparison**: By evaluating models with different interfaces on identical cases, the benchmark enables fair comparison across conditioning types.

3. **Living Benchmark**: The extensible skill library and self-improving evaluation loop address the evolving nature of world models.

## Conclusion

HarnessEval-W represents a paradigm shift in world model evaluation, moving from black-box scalar metrics to transparent, reasoning-based evaluation. The benchmark demonstrates that:
1. **Agentic evaluation** can align closely with human judgment while providing interpretable, verifiable results
2. **Adaptive testing** reveals capability distributions that static benchmarks miss
3. **The evaluation harness** can evolve with the models it measures, addressing the challenge of benchmark obsolescence

The authors propose three future directions:
1. **Test-time scaling** for agentic benchmarks, applying more compute to deeper analysis
2. **Scaling skill libraries** to cover increasingly complex world scenarios
3. **Recursively self-improving** agentic benchmarks that acquire new skills autonomously

The benchmark is open-sourced as a living system, inviting community contribution to grow new skills and evaluation cases as world models evolve.

---

_Markdown view of https://picx.dev/p/R5jC88, served by PicX — AI-generated visual whiteboard summaries of research papers._