Summary (Overview)

• Systematic study of world models as policy evaluators: The paper analyzes 7 video world models, 4 action representation paradigms, and over 324,000 simulated policy rollouts, introducing WMBench—a benchmark of paired real-world and world-model rollouts across 8 manipulation tasks.
• Three core insights: Evaluator quality is dominated by long-horizon, action-faithful rollout consistency; pretraining benefits from balancing general world knowledge with robot-specific controllability; architectural choices (action encoding, memory design, evaluator-focused post-training) strongly determine alignment with real-world behavior.
• Introduction of GigaWorld-1: A world model optimized for policy evaluation, built on Wan backbone with explicit pixel-aligned action control, hierarchical history memory, relative RoPE, and progressive multi-stage training. It improves evaluator-alignment metrics by 14.9% over competitive baselines (Cosmos-Predict2.5, Wan 2.2).
• VLM-assisted evaluation pipeline: A LoRA-tuned Qwen3-VL evaluator achieves near-perfect agreement with human ratings (87.80% exact, 99.16% adjacent, QWK=0.7349), enabling scalable outcome annotation.
• Open-source release: Code, models, datasets, and toolkits are fully open-sourced to advance scalable evaluation for embodied foundation models.

Introduction and Theoretical Foundation

Evaluating embodied robot foundation models is a critical bottleneck. Unlike large language models, which can be rapidly assessed via digital benchmarks, robotic policies require slow, costly real-world rollouts dependent on hardware and human supervision. This motivates using world models as surrogate policy evaluators—learned simulators that predict future observations conditioned on policy actions. However, the key properties that make a world model reliable for policy assessment remain poorly understood.

The paper formulates the problem: given a robot policy $\pi$ that outputs actions $a_t = \pi(o_t, s_t, l)$ and a world model $M_\theta$ that predicts future observations $\hat{o}_{t+1:t+H} \sim M_\theta(\cdot \mid o_{\le t}, s_{\le t}, a_{\le t}, l)$, the goal is to preserve the ranking and success prediction of real-world outcomes. The central alignment measure is the ranking correlation:

$$\rho = \text{Corr}\big(S_{\text{real}}(\pi), S_{\text{wm}}(\pi)\big)$$

where $S_{\text{real}}$ and $S_{\text{wm}}$ are empirical success rates from real and world-model rollouts, respectively. The paper systematically investigates three questions: (1) how to assess evaluator quality beyond video metrics, (2) how pretraining and data composition affect evaluator reliability, and (3) which architectural design choices matter most.

Methodology

WMBench Benchmark Construction

• Data sources: 2,989 paired trajectories across 8 tasks, comprising teleoperated real-world data and policy rollouts from GigaBrain checkpoints (50% each). Train-test split is episode-disjoint, outcome-balanced, and diversity-preserving.
• Data cleaning: Removal of corrupted/truncated videos, desynchronized clips, missing robot states, ambiguous outcomes, and near-duplicates.
• Large-scale rollout dataset: 324,000 world-model rollout segments from CVPR 2026 challenge submissions, human-annotated on a four-level World Model as Evaluator Score (WMES):
  • 3: correct outcome, high visual fidelity
  • 2: correct outcome, low fidelity
  • 1: incorrect outcome, high fidelity
  • 0: incorrect outcome, low fidelity

Evaluation Protocol (4 Steps)

1. Real-world policy data collection (multi-view videos, success labels)
2. World model training on train split; test episodes strictly held out
3. Closed-loop rollout: policy predicts action → world model predicts observation → fed back as next input
4. Metric calculation: automatic metrics (visual fidelity, geometry, semantics, motion) + WMES (human/VLM)

Model Architecture: GigaWorld-1

Built on Wan [1.3B / 5B] backbone with:

• Unified control injection: pixel-aligned action representation—EE pose maps for head view, ray maps for wrist views—concatenated and encoded as $Z_{\text{ctrl}} = E(C)$.
• Hierarchical history memory: first-frame anchor + short-, mid-, long-range memories to stabilize long-horizon rollouts.
• Relative RoPE: local temporal positions reset per autoregressive step to avoid position drift.
• Prompt transition via SLERP: spherical linear interpolation between prompt embeddings for smooth semantic transitions.
• Progressive training: Stage 1 (robot domain flow-matching), Stage 2 (autoregressive rollout learning), Stage 3 (optional scene LoRA), Stage 4 (DMD2 distillation for few-step inference).

VLM-Based Outcome Evaluation

LoRA-tuned Qwen3-VL-8B-Instruct with token-type-aware loss weighting (score token weight 8.0, rationale tokens weight 0.05–1.0). Predicts WMES on 0–3 scale with structured rationales (overall quality, instruction following, physical adherence).

Empirical Validation / Results

Key Findings from Question I (Evaluator Quality Assessment)

• Finding 1: Visual fidelity ($\rho=0.78$) and geometry ($\rho=0.71$) dominate WMES prediction. Subject Consistency ($\rho=0.88$) and Perspectivity ($\rho=0.86$) are strongest individual metrics.
• Finding 2: Degenerate metrics (Background Consistency $\rho=-0.45$, Photometric Consistency $\rho=-0.42$, Interaction Quality $\rho=-0.11$) mislead ranking—static videos score high on appearance but fail at evaluation.
• Finding 3: Long-horizon rollout quality (PSNR, FID, FVD over 40s) is essential; short-horizon generators like SVD, Cosmos, LTX suffer from viewpoint drift, identity collapse, and degradation.
• Finding 4: VLM evaluator achieves 87.80% exact agreement, 99.16% adjacent agreement, QWK 0.7349, Spearman 0.7574 on 5000+ videos.

Key Findings from Question II (Pretraining & Data)

• Finding 6: Transferable physical priors matter more than raw scale. Cosmos-Predict2.5 (2B, robot/Auto pretraining) outperforms larger general models (LTX 22B, Wan 2.2 5B).
• Finding 7: Broad physical videos (PhysData) provide best overall trade-off: +0.0490 average score over GigaData-only baseline, improving Photometric Consistency (+0.3074) while preserving task-relevant bias.
• Finding 8: Robot-specific data (AgiBot) improves embodiment fidelity (+0.0286 average) but introduces sharper trade-off: large drops in JEPA Similarity (-0.2426) and Trajectory Accuracy (-0.1084).

Key Findings from Question III (Model Design)

• Finding 9: Channel-concatenated control maps outperform cross-attention and ControlNet—Trajectory Accuracy 0.3528 vs. 0.1620 (cross-attention), 0.2566 (ControlNet).
• Finding 10: Persistent memory (first-frame anchor + hierarchical memory) is critical for long-horizon rollout: Wan 2.1+Mem. achieves PSNR 17.41 vs. 13.37 (Wan 2.1) at 32–40s interval, FVD 98.34 vs. 320.52.

Final GigaWorld-1 Evaluation

Table 9: Model architecture comparison (selected metrics, higher is better)

GigaWorld-1-Plus improves average by 11.6% over Cosmos-Predict2.5 and 14.9% over Wan 2.2 5B, with gains concentrated on evaluator-critical metrics (JEPA Similarity, Semantic Alignment, Trajectory Accuracy).

Closed-loop policy consistency: GigaWorld-1 better preserves real-world success/failure conclusions than challenge baselines, showing smaller Gen–Real deviation across 12 subtasks.

Theoretical and Practical Implications

• Redefining evaluation targets: The paper establishes that world-model evaluator quality cannot be reduced to video-generation metrics; the decisive property is agreement with real-world policy outcomes under iterative rollout.
• Design principles: Successful evaluator-oriented world models require:
  • Data: Balancing broad physical priors with robot-specific controllability
  • Action interface: Explicit, spatially aligned control representation (channel-concatenated maps)
  • Memory: Persistent history to prevent compounding error in long-horizon rollouts
  • Evaluation metrics: Subject Consistency, Perspectivity, and long-horizon PSNR/FID/FVD are strong proxies; appearance-stability metrics are misleading.
• Scalable evaluation pipeline: The VLM-assisted evaluator (LoRA-tuned Qwen3-VL) reduces annotation cost while maintaining high agreement with human raters, enabling large-scale policy iteration.
• Benchmark contribution: WMBench provides a standardized, open-source platform for studying world-model evaluators, with over 50,000 downloads on Hugging Face.

Conclusion

The paper presents a comprehensive roadmap for building world models as reliable policy evaluators. The central insight is that evaluator quality depends primarily on long-horizon, action-faithful consistency and transferable physical priors, not on short-term visual realism alone. These findings are instantiated in GigaWorld-1, which achieves state-of-the-art evaluator alignment (14.9% improvement over baselines) through explicit action encoding, hierarchical memory, and progressive training.

Limitations: WMBench covers only 8 task families (no mobile manipulation, dexterous in-hand tasks, or safety-critical autonomy); the study focuses on video-centric models; VLM evaluation still benefits from human verification in uncertain cases.

Future directions: Expanding the benchmark with more rollout data; developing more accurate metrics for outcome faithfulness and action-conditioned consistency; scaling model parameters while preserving evaluator-oriented design; extending to richer reward modeling, counterfactual evaluation, and certified uncertainty.