# GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation

> World model evaluator quality is dominated by long-horizon action-faithful rollout consistency and transferable physical priors, not short-term visual realism.

- **Source:** [arXiv](https://arxiv.org/abs/2607.02642)
- **Published:** 2026-07-08
- **Permalink:** https://picx.dev/p/QXd398
- **Whiteboard:** https://picx.dev/p/QXd398/image

## Summary

## Summary (Overview)

- **Systematic study of world models as policy evaluators**: The paper analyzes 7 video world models, 4 action representation paradigms, and over 324,000 simulated policy rollouts, introducing WMBench—a benchmark of paired real-world and world-model rollouts across 8 manipulation tasks.
- **Three core insights**: Evaluator quality is dominated by long-horizon, action-faithful rollout consistency; pretraining benefits from balancing general world knowledge with robot-specific controllability; architectural choices (action encoding, memory design, evaluator-focused post-training) strongly determine alignment with real-world behavior.
- **Introduction of GigaWorld-1**: A world model optimized for policy evaluation, built on Wan backbone with explicit pixel-aligned action control, hierarchical history memory, relative RoPE, and progressive multi-stage training. It improves evaluator-alignment metrics by 14.9% over competitive baselines (Cosmos-Predict2.5, Wan 2.2).
- **VLM-assisted evaluation pipeline**: A LoRA-tuned Qwen3-VL evaluator achieves near-perfect agreement with human ratings (87.80% exact, 99.16% adjacent, QWK=0.7349), enabling scalable outcome annotation.
- **Open-source release**: Code, models, datasets, and toolkits are fully open-sourced to advance scalable evaluation for embodied foundation models.

## Introduction and Theoretical Foundation

Evaluating embodied robot foundation models is a critical bottleneck. Unlike large language models, which can be rapidly assessed via digital benchmarks, robotic policies require slow, costly real-world rollouts dependent on hardware and human supervision. This motivates using **world models** as surrogate policy evaluators—learned simulators that predict future observations conditioned on policy actions. However, the key properties that make a world model reliable for policy assessment remain poorly understood.

The paper formulates the problem: given a robot policy $\pi$ that outputs actions $a_t = \pi(o_t, s_t, l)$ and a world model $M_\theta$ that predicts future observations $\hat{o}_{t+1:t+H} \sim M_\theta(\cdot \mid o_{\le t}, s_{\le t}, a_{\le t}, l)$, the goal is to preserve the ranking and success prediction of real-world outcomes. The central alignment measure is the ranking correlation:

$$
\rho = \text{Corr}\big(S_{\text{real}}(\pi), S_{\text{wm}}(\pi)\big)
$$

where $S_{\text{real}}$ and $S_{\text{wm}}$ are empirical success rates from real and world-model rollouts, respectively. The paper systematically investigates three questions: (1) how to assess evaluator quality beyond video metrics, (2) how pretraining and data composition affect evaluator reliability, and (3) which architectural design choices matter most.

## Methodology

### WMBench Benchmark Construction
- **Data sources**: 2,989 paired trajectories across 8 tasks, comprising teleoperated real-world data and policy rollouts from GigaBrain checkpoints (50% each). Train-test split is episode-disjoint, outcome-balanced, and diversity-preserving.
- **Data cleaning**: Removal of corrupted/truncated videos, desynchronized clips, missing robot states, ambiguous outcomes, and near-duplicates.
- **Large-scale rollout dataset**: 324,000 world-model rollout segments from CVPR 2026 challenge submissions, human-annotated on a four-level **World Model as Evaluator Score (WMES)**:
  - **3**: correct outcome, high visual fidelity
  - **2**: correct outcome, low fidelity
  - **1**: incorrect outcome, high fidelity
  - **0**: incorrect outcome, low fidelity

### Evaluation Protocol (4 Steps)
1. Real-world policy data collection (multi-view videos, success labels)
2. World model training on train split; test episodes strictly held out
3. Closed-loop rollout: policy predicts action → world model predicts observation → fed back as next input
4. Metric calculation: automatic metrics (visual fidelity, geometry, semantics, motion) + WMES (human/VLM)

### Model Architecture: GigaWorld-1
Built on Wan [1.3B / 5B] backbone with:
- **Unified control injection**: pixel-aligned action representation—EE pose maps for head view, ray maps for wrist views—concatenated and encoded as $Z_{\text{ctrl}} = E(C)$.
- **Hierarchical history memory**: first-frame anchor + short-, mid-, long-range memories to stabilize long-horizon rollouts.
- **Relative RoPE**: local temporal positions reset per autoregressive step to avoid position drift.
- **Prompt transition via SLERP**: spherical linear interpolation between prompt embeddings for smooth semantic transitions.
- **Progressive training**: Stage 1 (robot domain flow-matching), Stage 2 (autoregressive rollout learning), Stage 3 (optional scene LoRA), Stage 4 (DMD2 distillation for few-step inference).

### VLM-Based Outcome Evaluation
LoRA-tuned Qwen3-VL-8B-Instruct with token-type-aware loss weighting (score token weight 8.0, rationale tokens weight 0.05–1.0). Predicts WMES on 0–3 scale with structured rationales (overall quality, instruction following, physical adherence).

## Empirical Validation / Results

### Key Findings from Question I (Evaluator Quality Assessment)
- **Finding 1**: Visual fidelity ($\rho=0.78$) and geometry ($\rho=0.71$) dominate WMES prediction. Subject Consistency ($\rho=0.88$) and Perspectivity ($\rho=0.86$) are strongest individual metrics.
- **Finding 2**: Degenerate metrics (Background Consistency $\rho=-0.45$, Photometric Consistency $\rho=-0.42$, Interaction Quality $\rho=-0.11$) mislead ranking—static videos score high on appearance but fail at evaluation.
- **Finding 3**: Long-horizon rollout quality (PSNR, FID, FVD over 40s) is essential; short-horizon generators like SVD, Cosmos, LTX suffer from viewpoint drift, identity collapse, and degradation.
- **Finding 4**: VLM evaluator achieves 87.80% exact agreement, 99.16% adjacent agreement, QWK 0.7349, Spearman 0.7574 on 5000+ videos.

### Key Findings from Question II (Pretraining & Data)
- **Finding 6**: Transferable physical priors matter more than raw scale. Cosmos-Predict2.5 (2B, robot/Auto pretraining) outperforms larger general models (LTX 22B, Wan 2.2 5B).
- **Finding 7**: Broad physical videos (PhysData) provide best overall trade-off: +0.0490 average score over GigaData-only baseline, improving Photometric Consistency (+0.3074) while preserving task-relevant bias.
- **Finding 8**: Robot-specific data (AgiBot) improves embodiment fidelity (+0.0286 average) but introduces sharper trade-off: large drops in JEPA Similarity (-0.2426) and Trajectory Accuracy (-0.1084).

### Key Findings from Question III (Model Design)
- **Finding 9**: Channel-concatenated control maps outperform cross-attention and ControlNet—Trajectory Accuracy 0.3528 vs. 0.1620 (cross-attention), 0.2566 (ControlNet).
- **Finding 10**: Persistent memory (first-frame anchor + hierarchical memory) is critical for long-horizon rollout: Wan 2.1+Mem. achieves PSNR 17.41 vs. 13.37 (Wan 2.1) at 32–40s interval, FVD 98.34 vs. 320.52.

### Final GigaWorld-1 Evaluation

**Table 9: Model architecture comparison (selected metrics, higher is better)**
| Model | Size | Aesthetic ↑ | Image ↑ | JEPA ↑ | Semantic ↑ | Subject ↑ | Trajectory ↑ | AVG ↑ |
|-------|------|-------------|---------|--------|------------|-----------|--------------|-------|
| SVD | 1.5B | 0.2861 | 0.6497 | 0.6454 | 0.8411 | 0.8267 | 0.0926 | 0.5569 |
| Wan 2.2 5B | 5B | 0.3538 | 0.6980 | 0.5853 | 0.8789 | 0.8883 | 0.1643 | 0.5948 |
| Cosmos-Predict2.5 | 2B | 0.3491 | 0.7184 | 0.6781 | 0.8764 | 0.8747 | 0.1770 | 0.6123 |
| GigaWorld-1-Nano | 1.3B | 0.3538 | 0.6802 | 0.8911 | 0.8920 | 0.8600 | 0.3528 | 0.6717 |
| GigaWorld-1-Plus | 5B | 0.3534 | 0.6765 | 0.9337 | 0.8926 | 0.8883 | 0.3561 | **0.6834** |

GigaWorld-1-Plus improves average by 11.6% over Cosmos-Predict2.5 and 14.9% over Wan 2.2 5B, with gains concentrated on evaluator-critical metrics (JEPA Similarity, Semantic Alignment, Trajectory Accuracy).

**Closed-loop policy consistency**: GigaWorld-1 better preserves real-world success/failure conclusions than challenge baselines, showing smaller Gen–Real deviation across 12 subtasks.

## Theoretical and Practical Implications

- **Redefining evaluation targets**: The paper establishes that world-model evaluator quality cannot be reduced to video-generation metrics; the decisive property is **agreement with real-world policy outcomes under iterative rollout**.
- **Design principles**: Successful evaluator-oriented world models require:
  - **Data**: Balancing broad physical priors with robot-specific controllability
  - **Action interface**: Explicit, spatially aligned control representation (channel-concatenated maps)
  - **Memory**: Persistent history to prevent compounding error in long-horizon rollouts
  - **Evaluation metrics**: Subject Consistency, Perspectivity, and long-horizon PSNR/FID/FVD are strong proxies; appearance-stability metrics are misleading.
- **Scalable evaluation pipeline**: The VLM-assisted evaluator (LoRA-tuned Qwen3-VL) reduces annotation cost while maintaining high agreement with human raters, enabling large-scale policy iteration.
- **Benchmark contribution**: WMBench provides a standardized, open-source platform for studying world-model evaluators, with over 50,000 downloads on Hugging Face.

## Conclusion

The paper presents a comprehensive roadmap for building world models as reliable policy evaluators. The central insight is that evaluator quality depends primarily on **long-horizon, action-faithful consistency** and **transferable physical priors**, not on short-term visual realism alone. These findings are instantiated in GigaWorld-1, which achieves state-of-the-art evaluator alignment (14.9% improvement over baselines) through explicit action encoding, hierarchical memory, and progressive training.

**Limitations**: WMBench covers only 8 task families (no mobile manipulation, dexterous in-hand tasks, or safety-critical autonomy); the study focuses on video-centric models; VLM evaluation still benefits from human verification in uncertain cases.

**Future directions**: Expanding the benchmark with more rollout data; developing more accurate metrics for outcome faithfulness and action-conditioned consistency; scaling model parameters while preserving evaluator-oriented design; extending to richer reward modeling, counterfactual evaluation, and certified uncertainty.

---

_Markdown view of https://picx.dev/p/QXd398, served by PicX — AI-generated visual whiteboard summaries of research papers._