Summary (Overview)

• ABot-World-0 is an action-conditioned video world model that enables real-time, long-horizon interactive world rollout on a single desktop GPU (NVIDIA RTX 5090), achieving up to 16 FPS at 720P with 1.2 s action-to-first-frame latency and ~19 GiB peak VRAM.
• It uses raw keyboard inputs as a unified control interface for both scene roaming (camera navigation) and third-person character interaction, with reference-character memory for identity persistence.
• A multi-source data infrastructure integrates AAA games, simulation engines (UE, 3DGS), and internet videos via the agent-driven collection system WorldExplorer, with 14 deterministic quality checks and VLM-based annotation.
• The training pipeline progressively distills a bidirectional teacher into a causal student via teacher forcing, ODE distillation, and a novel LongForcing stage that aligns long student self-rollouts with an extended-horizon teacher to mitigate autoregressive drift.
• The deployment stack co-designs a lightweight VAE decoder, efficient attention (SageAttention2), Fast-RoPE, memory-aware scheduling, and low-bit DiT inference (FP8/MXFP6/MXFP4), making interactive world modeling practical on consumer hardware.

Introduction and Theoretical Foundation

The paper addresses the challenge of building a persistent generative environment where user actions continuously change the visual state, and the world remains coherent over long interactions. This goes beyond static video generation: it requires controllability, state persistence, rollout stability, low latency, high throughput, and memory efficiency.

Existing world models (Genie, GameNGen, Oasis, Cosmos, etc.) tackle parts of this agenda but face four coupled bottlenecks:

1. Data: obtaining broad, temporally coherent video with reliable action supervision.
2. Control: representing user intent for both camera navigation and embodied character control.
3. Drift: preventing generated history from degrading as it becomes input for subsequent predictions.
4. Deployment: running the full generation-and-decoding stack at interactive speed on practical hardware.

The paper treats these as a full-stack systems challenge rather than a single generative objective. The theoretical basis rests on action-conditioned video prediction as a world model: given past visual observations, future actions, and multimodal conditions (text, reference images), the model predicts future video chunks autoregressively.

Methodology

1. Data Infrastructure

Three complementary sources are used:

• AAA game data: captured via WorldExplorer with ground-truth API actions, synchronized RGB and poses.
• Simulation data: Unreal Engine and ABot-3DGS (proprietary photorealistic reconstructions) with procedural or real-world trajectory imports.
• Internet videos: pose-estimated pseudo-labels from diverse real-world footage.

WorldExplorer is an agent-driven collection system with:

• Navigation agent using multi-phase goal selection.
• Parallel capture pipeline (synchronized multimodal recording).
• Task template system for behavioral coverage.
• Training-feedback-driven closed-loop: monitoring model weaknesses, diagnosing failure modes, and adaptively rebalancing collection ratios.

Quality filtering applies 14 deterministic checks across six dimensions (file integrity, visual validity, geometric consistency, game state correctness, action-label alignment, metadata quality) plus VLM-based semantic screening.

Annotation produces:

• Unified action labels (8-dimensional multi-hot for W/A/S/D and I/J/K/L).
• VLM-generated scene descriptions (without camera motion info).
• Semantic tags (weather, time, viewpoint, etc.).
• Person identity thumbnails for third-person rollouts.

2. Model Architecture and Training

Problem formulation: The world model predicts future video chunks conditioned on visual history, actions, and multimodal conditions:

$$p_\theta(v_{t:t+L-1} | v_{0:t-1}, a_{t:t+L-1}, c) \qquad (1)$$

Bidirectional teacher training (Section 4.2):

• Fine-tunes a pretrained Wan2.2 backbone with action control injection.
• Actions are packed per 4 frames (to match VAE temporal patch size): $\tilde{a}_\tau = \text{Concat}(a_{4\tau-3}, a_{4\tau-2}, a_{4\tau-1}, a_{4\tau})$ (Eq. 6), yielding 32-dimensional tokens.
• Additive injection via Action Control Adapter $F_\psi$: $\hat{z} = \text{PatchEmbed}(z) + F_\psi(\tilde{a}_{1:T/4})$ (Eq. 7).
• Identity-preserving conditioning: reference-character images are encoded as memory tokens with negative temporal RoPE indices, attended asymmetrically by video tokens.

Causal student distillation (three stages):

1. Teacher Forcing (Stage 1): Converts bidirectional attention to causal masking. Student predicts $p^\text{causal}_\theta(v_{t:t+L-1} | v_{0:t-1}, a_{t:t+L-1}, c)$ using clean history and noisy target chunks.

2. ODE Distillation (Stage 2): Freezes Stage 1 causal model. Distills few-step prediction by learning the clean endpoint $z^c_0$ from intermediate latent $z^c_s$ along the probability-flow ODE:

$$\mathcal{L}_\text{ODE} = \mathbb{E}_{s, z^c_s, C_t} \left[ \| f_\theta(z^c_s, s, C_t) - \text{sg}(z^c_0) \|_2^2 \right] \qquad (11)$$

where $C_t = (v_{0:t-1}, a_{t:t+L-1}, c)$.

3. LongForcing (Stage 3): Extends distribution matching to long student self-rollouts. Uses an extended-horizon bidirectional teacher to provide corrective supervision via DMD (Distribution Matching Distillation) on self-generated trajectories, reducing accumulated autoregressive drift.

3. Deployment Co-Design

Real-time inference on a single RTX 5090 requires joint optimization of:

• Lightweight VAE decoder (LightVAE) to reduce decoding time and memory.
• Memory-aware scheduling (module swapping) to keep peak VRAM low.
• Fast-RoPE for efficient temporal positional encoding in streaming.
• Low-bit DiT inference (FP8 default, also MXFP6/MXFP4) for compute and memory savings.
• SageAttention2 for efficient attention kernels.
• Bounded KV caching with rolling eviction and optional cache quantization.

Table 2 shows the system-level breakdown across optimization variants at 1280×704 resolution on a single RTX 5090:

Each chunk produces 12 decoded video frames. The final operating envelope: up to 16 FPS, 1.2 s latency, <19.3 GiB VRAM.

Empirical Validation / Results

WorldRoamBench Evaluation

Table 3 compares ABot-World-0 (5B model) against Genie 3, HappyOyster, LingBot-World, and HY-World 1.5 on key sub-dimensions:

ABot-World-0 achieves best or second-best in all metrics, with particularly strong action fidelity and mechanics.

LongForcing Ablation (60-second rollouts)

Figure 10 shows that LongForcing maintains higher HPSv3 aesthetic scores and lower artifact metrics (high-saturation pixel ratio, perceptual blur, patch repeat ratio) compared to a Causal-Forcing-style baseline, especially in the second half of the rollout. This demonstrates reduced visual error accumulation.

Qualitative Results

• Hour-scale rollouts (Figure 5): five independent 1-hour rollouts show sustained scene coherence and controllability.
• Day-scale stress tests (Figures 6, 7): 24-hour rollouts preserve recognizable structure and active motion without collapse.
• Out-of-domain generalization (Figure 8): unified action interface works across diverse scenes and characters not seen in training.
• Physical interactions (Figure 9): emergent plausible effects – object pushing, water disturbances, snow footprints, wall blocking, collision without interpenetration.

Theoretical and Practical Implications

• Explicit actions simplify conditioning: Discrete keyboard inputs provide a natural, bounded control interface that aligns with user intent and avoids the complexity of latent actions or calibrated camera poses.
• Long-horizon drift as distribution shift: LongForcing treats drift as a distribution-matching problem rather than a local error-correction one. By supervising on student self-rollouts with an extended-horizon teacher, the model learns to stay within plausible dynamics without excessive anchoring.
• Real-time interaction is a systems problem: Few-step diffusion alone is insufficient; full-stack co-design (VAE, attention, memory, scheduling, quantization) is necessary to achieve interactive frame rates on consumer hardware.
• Practical deployment: ABot-World-0 demonstrates that a single desktop GPU can serve as a persistent, controllable world simulator, opening doors for interactive content creation, game prototyping, agent learning, and embodied-AI research without requiring cloud infrastructure.

Conclusion

ABot-World-0 delivers a unified, action-conditioned video world model that runs in real time on a single NVIDIA RTX 5090. Key innovations include:

• Multi-source data infrastructure with training-feedback-driven collection.
• Progressive bidirectional-to-causal distillation with LongForcing for long-horizon stability.
• Full-stack inference co-design achieving up to 16 FPS at 720P with 1.2 s latency and ~19 GiB VRAM.

The model shows competitive performance on WorldRoamBench, maintains coherence over day-scale rollouts, generalizes to out-of-domain scenarios, and exhibits emergent physical plausibility. Future work includes richer action conditioning, multi-scale LongForcing, persistent scene memory, and broader hardware support. ABot-World-0 represents a practical step toward local, interactive world simulators for creation, learning, and embodied intelligence.