Agentic World Models and Reinforcement Learning with Human-Engine Verification (RLHEV)

Summary

This paper introduces a paradigm shift for spatial intelligence and world models by applying the successful recipe of code agents—iterative execution feedback plus human verification—to 3D world generation. The authors propose Agentic World Models (AWM) trained via Reinforcement Learning with Human-Engine Verification (RLHEV), a framework that treats game development as a grounded, verifiable feedback loop. Key contributions include:

• RLHEV training paradigm: Combines dense, low-cost engine verification (collision, navmesh, physics) with sparse human acceptance judgments to train world models, analogous to compiler-plus-developer feedback in coding.
• UnitySceneBench (200-example evaluation): Demonstrates that RLHEV-trained models achieve state-of-the-art asset classification and generation quality.
• Out-of-distribution generalization: Transfer learning from Unity to Unreal and Godot improves performance (0.25 → 0.75 for distribution shift, 0.25→0.35 and 0.25→0.46 for cross-engine).
• Embodied policy gains: RLHEV-augmented training improves downstream policy performance by +0.79% on R2R, +9.96% on Gymnasium MuJoCo, and +48.43% on D4RL Gym-MuJoCo.
• Unified World Development Protocol (UWDP): A structured trajectory format that records edit-propose-check-accept/reject traces, making game development a self-improving data engine for spatial intelligence.

Introduction

The paper identifies a fundamental asymmetry between code intelligence and spatial intelligence. Code agents succeed because code is executable: compilers, tests, and linters provide dense, low-cost feedback, and developers add final acceptance judgments. This dual signal enables reinforcement learning from verifiable feedback (RLVR) and continuous post-training improvement. Spatial intelligence—generating 3D scenes, virtual environments, and interactive worlds—currently lacks such a loop. Generated outputs are typically scored by fuzzy proxies (JSD, FVD, CLIP similarity) or human preference models, which are expensive, noisy, and not grounded in physical or functional correctness.

The authors argue that game development naturally provides the missing verification channel: modern engines like Unity, Unreal, and Godot can execute proposed worlds and return structural checks (collision, navmesh, physics stability, script execution). Meanwhile, human developers provide a global acceptance judgment. The paper thus frames game development as a scalable, "grounded" data source for spatial intelligence post-training.

Introduction and Motivation

The Scaling Bottleneck

While language models benefit from verifiable feedback (e.g., code execution, math solvers), spatial intelligence suffers from fuzzy supervision: quality is assessed via CLIP scores, FVD, and MLLM-as-judge ratings—metrics that do not reflect physical correctness. The authors note three limitations of current spatial intelligence:

1. Unverifiability of proxy metrics: CLIP similarity or MLLM-as-judge scores do not measure functional correctness—whether objects collide properly, floors are navigable, or doors open.
2. Costliness of human annotation for 3D data: Unlike text, 3D scenes require expensive scans, manual cleanup, and alignment; no cheap automatic verifier exists.
3. A mismatch between appearance and function: Generated worlds may look plausible but fail structurally (e.g., a chair intersecting a wall).

The crucial insight: game engines already provide dense, low-cost verification (collision queries, navmesh reachability, script execution, playability checks), and game developers provide sparse, high-quality acceptance decisions. This dual-verification loop turns world-building into a self-improving feedback engine.

Introduction

The Scaling Bottleneck for Spatial Intelligence

Recent world-model research treats spatial intelligence mostly as a scaling problem: collect more scraped video, train bigger models, spend more compute on 3D scans. The authors argue this recipe is fundamentally inefficient for spatial intelligence because the field lacks a cheap, dense, and grounded verification channel.

The "Unverifiability Tax": Without reliable verification, progress depends heavily on brute-force scaling of data, compute, and human annotation, which is far more expensive than pure language modeling. The paper draws a parallel to code agents: before execution-based feedback (RLVR-style), code models plateaued; after "code is executable" verification (tests, compilers), they improved dramatically McKnight et al. 2025, Sun et al. 2025, Huang et al. 2025.

The Key Insight: Game development already contains both halves of the verification loop:

1. Engine verification (dense, low-cost, grounded): Physics collisions, navmesh reachability, script errors, bounded playability probes.
2. Human acceptance (sparse, high-level): Developer decides if a scene "fits the design intent" (mood, maintainability, product goal).

Dual-verification hypothesis: Combining these two signals—engine-level correctness and human-level acceptance—provides a training signal that approaches the "executable-and-functional" property of code, which is missing in plain video or static 3D generation.

Methodology — RLHEV and the UWDP

Training Objective

RLHEV is framed as constrained selection over candidate world edits:

$$\max_{y \in \mathcal{Y}} \; U_H(x, y, h)$$

where $U_H$ is the human acceptance judge, implicitly marginalized with engine checks.

The Full Pipeline (Engine + Human Feedback)

The full RLHEV reward is a weighted combination:

• Engine rewards: Dense structural checks (collision validity, navmesh connectivity, settled physics, bounded playability) derived from Unity's physics/debug APIs.
• Human rewards: Binary "accept/reject" labels from developers reviewing the scene.

In the paper's experiments, the full RLHEV reward is: $0.35 \times \text{engine} + 0.65 \times \text{human}$.

The generic reward formulation is:

$$U_t \propto w_E \cdot \mathbf{1}[\text{engine checks pass}] + w_H \cdot \mathbf{1}[\text{human accepts}]$$

Training Protocol (UWDP): Each scene edit produces a structured training instance: (scene_state, proposed_edit, engine_check_results, developer_acceptance, trace). These traces are accumulated and used for post-training, analogous to how successful GitHub pull requests train code models.

---

Methodology

The Agentic World Model (AWM)

The system is a multimodal Agentic World Model with:

• Unified backbone (multimodal LLM) that processes both images (scene renders) and text (edit proposals, error logs).
• Policy head for environment interaction: proposes edits, reads engine outputs (collision reports, navmesh failures), and iterates.

The RLHEV Objective

The authors formalize a constrained selection problem:

$$\max_{y \in \mathcal{Y}} U_H(x, y, h)$$

subject to engine verification constraints ℰ(x, y) = 1 (well-formed, no collision, reachable) and human acceptance $\mathcal{H}(x, y, h) = 1$ (design intent satisfied).

Reward decomposition:

$$R_{total} = \lambda_E \cdot R_{engine} + \lambda_H \cdot R_{human}$$

with $\lambda_E = 0.35$, $\lambda_H = 0.65$ in the reported full model. Engine rewards are dense and structural (collision validity, navmesh connectivity, bounded playability probes); human rewards are sparse acceptance/rejection labels.

The RLHEV reward:

$$\max_{\theta} \mathbb{E}_{(x, h) \sim \mathcal{D}} \left[ \lambda_H \cdot U_H(x, y, h) + \lambda_E \cdot U_E(x, y) \right]$$

Methodology

Unified World-Development Protocol (UWDP)

The method is built around a structured trajectory protocol capturing the full edit/review loop:

1. Engine-checkable dense structural signals: Auto-generated meshes and scripts are submitted to engine checks (collision, navmesh reachability, script execution).
2. Human acceptance signals: Developers provide binary accept/reject per edit, combined with repair traces.
3. Training signal: $\text{reward} = \lambda_H U_H(x, y, h) + \lambda_E U_E(x, y)$, where $U_H$ is human acceptance and $U_E$ is engine-derived structural validity. RLHEV optimizes against this combined signal via a proximal policy optimization-style update similar to RLVR-style reasoning models (Huang et al., 2021; Shao et al., 2025).

RLHEV Core Formulation

The paper formalizes the Human-Engine constrained RLHF objective. Following the RLVR/Qwen-style approach, the optimal reward is the clipped advantage of a policy under human-engine feedback:

$$\text{RLHEV} = \mathbb{E}_{(x, y) \sim \pi_{\theta}} \left[ \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)} \cdot A_{HE}(x, y) \right] - \beta \cdot \mathbb{D}_{\mathrm{KL}}\left( \pi_\theta \| \pi_{\text{ref}} \right)$$

where the advantage $A_{HE}$ combines engine verification with human acceptance:

$$A_{HE}(x, y) = \text{Normalize}\left( \lambda_e \cdot R_{\text{engine}}(x, y) + \lambda_h \cdot R_{\text{human}}(y) \right)$$

with $\lambda_e = 0.35$ (engine) and $\lambda_h = 0.65$ (human).

Training recipe: Standard supervised fine-tuning on human-curated data followed by RLVR-style training with the dual-reward RLHEV.

---

Introduction

The paper argues that world models (spatial intelligence) need not just more data but more verifiable feedback channels. The authors draw heavily on the analogy to coding agents: code models improved dramatically when "execution feedback" became available. The paper's premise is that game engines provide the spatial equivalent of code execution.

Why game development?

• Groundedness: A collider, navmesh, and rigid-body check are physical facts, not human preferences.
• Low-cost clustering: Engine checks are cheap to compute repeatedly (unlike human annotation).
• Dense structure: Engine checks give many per-edit signals (collision OK, reachability OK, script OK, etc.).
• Sparse acceptance: The developer provides the final "does this feel right?" judgment.

Core Contributions:

1. Proposal of RLHEV (Reinforcement Learning with Human-Engine Verification) as a post-training paradigm.
2. Introduction of Agentic World Models (AWM) that propose, edit, and repair scenes using engine-plus-human feedback.
3. Design of the Unified World-Development Protocol (UWDP) for recording structured training trajectories from game development.
4. A benchmark: UnitySceneBench (200 Unity scene-edit examples).
5. Empirical evidence that RLHEV-augmented training improves downstream embodied policies.

---

Methodology

Agentic World Model (AWM)

AWM is trained autoregressively on multimodal "world-edit" traces: given a current scene graph, the model proposes a transformation (e.g., "move object X here", "add ramp here", "fix missing collider"). Each edit is applied in the engine, which returns:

• Visual render (camera snapshots)
• Engine check results (collision, navmesh, playability probe)

The human then accepts or rejects the edit to a final dataset. The AWM learns to maximize:

• the probability of edits that are both engine-valid and human-accepted.

Engine checks (dense, cheap)

• Collision: Do any two colliders intersect?
• Physics: Does an object rest stably or drift/fly?
• Navmesh reachability: Can a navigation agent path from A to B? Is the destination reachable?
• Script errors / soft-locks: Does any gameplay script throw an exception or enter a dead state?
• Bounded playability probe: Can a scripted agent finish a "mini-task" (e.g., walk to a target) in under T steps?

Human check (sparse, expensive, but final)

• "Does this scene satisfy the design intent?" (yes/no)
• Applied at episode level, rarely, per asset-acceptance decision.

Reward combination

$$r_t = w_H \cdot r_{human} + w_E \cdot r_{engine}$$

with $w_H = 0.65$, $w_E = 0.35$.

---

Algorithm (implicit, from text)

• Generate N candidate scene edits for a task description (sampling).
• Apply each to the engine in parallel.
• Compute the engine score from colliders/navmesh/playability probes.
• Present the candidate to the human reviewer (for final acceptance).
• Filter and store accepted traces; perform policy-gradient update on those traces.
• Repeat with a fixed data budget (e.g., 720 training instances in UnitySceneBench).

---

Empirical Validation

Benchmark: UnitySceneBench: 200 scene-edit tasks covering asset placement, room structuring, object collision resolution, and navmesh-based path adjustments. Training uses up to 720 human-curated "edit traces."

Evaluation axes

1. Asset classification (understanding), see Figure 4.
2. Generation quality: Can the model propose new scene edits that are both engine-valid and human-accepted?
3. Embodied agent performance: Does training a world model with RLHEV improve downstream embodied agents on R2R, Gymnasium, D4RL?

Key findings:

• Full RLHEV (human+engine reward = 0.35/0.65 weighting) achieves the highest asset classification score, beating the best non-full baseline by +0.45 primary score and +0.23 accuracy/balanced accuracy (as measured in UnitySceneBench).
• At larger training budgets (720 instances), Full RLHEV reaches 0.8093 (asset classification), far above the best baseline 0.7822.
• Generalization: Pretraining on source + adaptation to target raises OOD performance from 0.25 (scratch) to 0.75.
• Cross-engine: Unity-trained checkpoints fine-tuned to Unreal (0.25→0.35) and Godot (0.25→0.75) show positive transfer.

Equation: RLVR-style loss with engine + human rewards

RLHEV reward:

$$R_{\text{RLHEV}}(s, a) = \lambda_{\text{eng}} \left[ \mathbb{1}_{\text{collision-free}} + \mathbb{1}_{\text{navmesh-reachable}} + \mathbb{1}_{\text{no-script-error}} \right] + \lambda_{\text{human}} \cdot \text{Accept}(s, a)$$

Weighted as 0.35 engine + 0.65 human in the full model.

---

Empirical Validation (UnitySceneBench)

---

Experiments

UnitySceneBench Setup

• Domain: Editing Unity 3D scenes via natural-language commands and asset edits.
• Training data: 720 scene-edit instances, each with: initial scene, edit command, applied edit diff, engine feedback (collision/navmesh/script), human accept/reject label.
• Test: 200 held-out scene-edit tasks requiring (a) classification: accept/reject a proposed edit assembly;
  (b) generation: propose the correct edit.

Results

1. Understanding: RLHEV (full dual reward) beats engine-only RLVR and CLIP-based classification baselines. Best score on classification: Full RLHEV, improving both accuracy and balanced accuracy vs. all baselines (Fig 3).
2. Generation: On up to 640 training instances, RLHEV achieves 0.8107 generation quality vs. 0.7934 for the best baseline (Engine-only RLVR), with +2% improvement. Notably, RLHEV with 320 instances outperforms all baselines trained on 640, confirming sample-efficiency.
3. Out-of-distribution (cross-engine): Transfer learning with RLHEV dramatically improves domain gaps, e.g., selecting/editing assets in Unreal and Godot without fine-tuning yields large gains.

---

Generalization & Analysis

Embodied Policy Improvements

Using RLHEV-augmented training data improves:

• R2R (Room-to-Room navigation): +0.79% success
• Gymnasium MuJoCo (continuous control): +9.96% average return
• D4RL Gym-MuJoCo: +48.43% average return

These results suggest that training with engine+human verification makes the data distribution more aligned with what downstream policies need.

Cost analysis

Engine checks on 200 scenes: ~0.3 seconds per check (collision) to ~1.1 seconds (bounded-playability probe). Estimated 50–200× cheaper than full manual evaluation, highlighting "verifiable and cheap" as the core benefit.

---

Related Work

• Code RLVR: McKnight et al. 2025; Sun et al. 2025; Huang et al. 2025 — the inspiration for RLHEV.
• Agentic world models: WorldCoder (executable agent worlds), VoxelCraft-style code generation, but without engine verification. The paper is the first to use engine+human dual verification for world-model RL.
• World-model RL: Contrasts with works that learn world models purely from video (e.g., DreamerV3) — those lack grounded verification.

---

Conclusion

The authors propose replacing "more data, more compute" with "better feedback infrastructure" for spatial intelligence. By turning 3D asset editing into a game-engine-grounded, human-verified pipeline and training with RLHEV, world models achieve better 3D understanding and cross-engine generalization, and they provide effective training signal for embodied policies Secret of RLVR. The core claim: game development is the best available "executable feedback" channel for 3D worlds, mirroring how coding-execution feedback unlocked code intelligence.

Future direction: Scaling to larger playable worlds (navmesh + playability probes) where engine checks become the scalable judge and human acceptance becomes a sparse fine-tuning signal.

Multi-source References

This synthesis is based on the paper "RLHEV: Reinforcement Learning with Human-Engine Verification for Agentic World Models" (project page: rlhev.agentic-world-model.org). The authors maintain that for spatial intelligence to progress, the community must adopt verifiable, executable world-feedback similar to code-execution feedback. The project page includes code, benchmark, and model checkpoints.

References cited: McKnight et al. 2025 (UnitySceneBench), Singh et al. 2024 (Genie 3), Cho et al. 2022 (MuJoCo tasks), Huang et al. 2025 (RLVR), Sun et al. 2025 (D4RL extensions). URL: https://rlhev.agentic-world-model.org.