AlayaWorld: Interactive Long-Horizon World Modeling – Full Technical Report

Summary (Overview)

• AlayaWorld is an interactive, long-horizon video world model that generates 24 fps video at 540p and 720p using a 15B video diffusion transformer (DiT). It synthesizes the world autoregressively in short latent chunks under continuous camera trajectories and dynamically switchable text prompts.
• Bounded visual context combines a persistent sink frame, compressed temporal history (6 frames), geometry-aligned spatial memory (up to 10 rendered cache frames), and a nearby frame, enabling constant per‑chunk compute and arbitrarily long roll‑outs.
• Anti‑drift training uses corrupted histories (Helios drift simulation) and an error bank that replays the model’s own prediction residuals, teaching the model to recover from imperfect context during long‑horizon generation.
• Discrete autoregressive distillation (Distribution‑Matching Distillation + self‑forcing++ + consistency distillation) reduces inference from approximately 30 sampling steps to 4 steps per chunk while preserving the full control and memory stack.
• On iWorld‑Bench, AlayaWorld achieves the best overall performance across generation quality, trajectory following, and memory ability, with particularly strong gains in brightness consistency, sharpness retention, and memory symmetry.

Introduction and Theoretical Foundation

The paper argues that turning a video generator into a genuinely interactive world model requires four tightly coupled capabilities:

1. Interaction – accurate response to camera trajectories and evolving user intent.
2. Consistency – preservation of spatial structure and visual identity across viewpoint changes and delayed revisits.
3. Stability – long autoregressive roll‑outs without accumulating blur, illumination shifts, or geometric drift.
4. Efficiency – low generation latency and rapid response to new control signals.

These capabilities cannot be addressed independently: broader interaction makes consistency harder to preserve, longer roll‑outs amplify residual errors, and aggressive acceleration may compromise visual stability.

AlayaWorld is built on top of the LTX‑2.3 multimodal model (22B parameters, with the audio module removed, leaving a ~13B video DiT). The paper formulates generation as a causal factorization in the VAE latent space:

$$p_\theta(z_{1:N} \mid \pi_{1:N}, y_{1:N}) = \prod_{i=1}^{N} p_\theta(z_i \mid z_{<i}, \pi_{\le i}, y_i), \tag{1}$$

where each chunk $z_i$ (4 latent frames) is generated conditioned on past latents, camera trajectory $\pi_{\le i}$, and an optional chunk‑level text prompt $y_i$.

Methodology

Data Curation

The training corpus mixes real‑world captures (for photorealism and geometry) with synthetic renderings (for scalable camera and action controllability). The final corpus contains 222 147 clips from seven sources, including two internally curated sources (MUGEN and GameVerse). Table 1 (from the paper) summarizes the data composition:

A unified, stage‑based filtering pipeline (technical validation, photometric, shot‑boundary, motion analysis, text/interface suppression, person control, and pose stability) ensures quality. Captions are annotated with a two‑level schema: video‑level global attributes and segment‑level temporal tracks (subject motion, environment motion, static scene, camera description). The separation between subject and camera tracks is crucial for distinguishing, e.g., walking subject vs. dolly‑in camera.

Model Architecture

The backbone is a single self‑attention DiT. For chunk $i$, the input token sequence is:

$$S_i = [\underbrace{s}_{\text{sink}}; \underbrace{h_i}_{\text{temporal memory}}; \underbrace{g_i}_{\text{spatial memory}}; \underbrace{n_i}_{\text{nearby/I2V}}; \underbrace{z_i^\tau}_{\text{target}}], \tag{2}$$

processed by full (non‑causal) self‑attention.

• Sink $s$ – a single clean latent frame at RoPE position 0, held fixed across all chunks as a global identity anchor.

• Temporal memory $h_i = H_\phi(w_i)$ – a compressed embedding of the last $L=6$ latent frames, computed by a trainable history‑compression module.

• Spatial memory $g_i$ – geometry‑aligned rendering of past views into the current view. Up to 10 previous frames are retrieved by greedy maximum‑coverage selection, warped into the target view via forward splatting:

  $$u' = \pi_i \left( \pi_j^{-1}(u, D_j(u)) \right), \tag{3}$$

  and fused with a coverage mask.

• Nearby / I2V condition $n_i$ – the single most‑recent latent frame, carrying full‑resolution frame‑to‑frame continuity.

Training Stages

Stage 1 – Bidirectional Pre‑training: Full‑parameter fine‑tune of the LTX‑2.3 backbone on world‑modeling data at 24 fps (540p/720p) under a mix of image‑, video‑, and text‑conditioned objectives. An adaptive sigma‑shift schedule scales the flow‑matching timestep shift with clip length.

Stage 2 – Autoregressive Training: Two phases:

• History pre‑training (backbone frozen, LoRA adapter): trains the history‑compression module $H_\phi$ using a masked history objective:

  $$\mathcal{L}_{2a} = \mathbb{E}\left[ \left\| v_\theta\left( z_\Omega^\tau, \tau \mid H_\phi(\tilde{z}) \right) - (\epsilon - z_\Omega) \right\|_2^2 \right], \quad z_\Omega^\tau = (1-\tau)z_\Omega + \tau\epsilon, \tag{4}$$

  where $\tilde{z}$ is a perturbed history window.

• Full‑stack fine‑tune: trains the backbone, history module, camera‑control module (AdaLN conditioned on Fourier‑embedded relative pose increments), spatial memory, and next‑forcing head:

  $$c_{\text{cam}} = \text{MLP}\left( \bigoplus_{k=1}^{6} \text{PE}(\Delta\pi_k) \right), \quad e \leftarrow e + c_{\text{cam}}, \tag{5}$$ $$\mathcal{L}_{\text{nf}} = \left\| f_\psi(F, z_0^+, \tilde{\tau}) - (\epsilon - z_0^+) \right\|_2^2, \quad \mathcal{L} = \mathcal{L}_{\text{flow}} + 0.5\mathcal{L}_{\text{nf}}. \tag{6}$$

Anti‑drift training: Combines Helios drift simulation (additive noise, down/up‑sampling blur, saturation shift) and an error bank that replays the model’s own reconstruction residuals $\delta = \hat{z}_0 - z_0$ into the context.

Stage 3 – Post‑Training Acceleration: Distills the many‑step teacher into a 4‑step student using:

• Distribution‑Matching Distillation (DMD) – matches student distribution to teacher via score‑difference gradient: $$\nabla_\theta D_{\text{KL}}(p_{\theta,\tau} \parallel p_{\text{data},\tau}) = -\mathbb{E}\left[ \left( s_{\text{real}}(\hat{z}_i^\tau, \tau \mid c_i) - s_{\text{fake}}(\hat{z}_i^\tau, \tau \mid c_i) \right) \frac{\partial \hat{z}_i}{\partial \theta} \right], \tag{7}$$ with both scores served by the same backbone via LoRA swapping.
• Self‑forcing++ – student rolls out its own multi‑chunk trajectories, closing the train/inference gap.
• Consistency distillation – enforces trajectory‑invariant outputs against an EMA target: $$\mathcal{L}_{\text{cm}} = \mathbb{E}\left[ d\left( G_\theta(z_i^\tau, \tau \mid c_i), G_{\theta^-}(z_i^{\tau'}, \tau' \mid c_i) \right) \right], \quad \tau' < \tau. \tag{8}$$

Empirical Validation / Results

Evaluation Setup

AlayaWorld is evaluated on iWorld‑Bench across three dimensions: Generation Quality, Trajectory Following, and Memory Ability. All results are from the distilled autoregressive model at 480p resolution with 4 sampling steps per chunk. Comparisons include Cosmos, HunyuanVideo‑1.5, WAN 2.2, YUME 1.5, Matrix‑Game 2.0, and HY‑World 1.5.

Quantitative Results

Key findings:

• AlayaWorld substantially outperforms competitors in brightness consistency (0.9492 vs. next best 0.8051) and sharpness retention (0.8361 vs. 0.6634), demonstrating effective mitigation of visual drift and illumination fluctuations.
• Trajectory accuracy (0.7985) and memory symmetry (0.8871) lead all methods, validating the spatial and temporal memory mechanisms for stable long‑horizon interaction.
• Despite using only 4 sampling steps, AlayaWorld remains competitive in image quality.

Qualitative Results

• Camera‑controllable navigation – smooth viewpoint changes under diverse translation and rotation commands while preserving coherent scene geometry.
• Prompt‑driven actions – newly specified actions (e.g., spell casting, combat) emerge naturally in subsequent chunks while existing scene context remains coherent.
• Consistent world generation – under leave‑and‑return trajectories, the model preserves scene layout, object identity, and textures.
• Long‑horizon stability – over 60‑second roll‑outs, limited accumulation of blur, illumination shifts, color drift, and structural degradation.

Theoretical and Practical Implications

Theoretical contributions:

• The paper formalizes interactive world modeling as a causal autoregressive factorization with bounded visual context, showing that interaction, consistency, stability, and efficiency can be jointly addressed.
• The discrete distillation formulation (DMD + self‑forcing++ + consistency distillation) avoids Jacobian‑vector‑product computation while maintaining the full control and memory stack, offering a practical acceleration method for autoregressive world models.
• The anti‑drift training strategy (Helios simulation + error bank) is a principled way to teach the model to recover from its own accumulated errors, directly addressing the fundamental challenge of long‑horizon stability.

Practical implications:

• AlayaWorld provides an open‑source, full‑stack foundation for interactive video world models, with code and pretrained models available.
• The 4‑step inference at 24 fps enables near‑real‑time interaction, making it suitable for applications in gaming, virtual reality, simulation, and content creation.
• The model supports both controllable navigation and open‑ended semantic actions, bridging the gap between video generation and interactive simulation.

Conclusion

AlayaWorld is an interactive long‑horizon video world model that integrates four tightly coupled capabilities (interaction, consistency, stability, efficiency) within a unified autoregressive framework. Its bounded visual context (sink, temporal memory, spatial memory, nearby frame) supports controllable navigation and persistent world memory. Drift‑aware training and discrete few‑step distillation enable stable long‑horizon generation and low‑latency interaction. On iWorld‑Bench, AlayaWorld achieves the best overall performance, with particularly strong gains in brightness consistency, sharpness retention, trajectory accuracy, and memory symmetry.

Limitations and future work:

• The model represents the world primarily through visual observations, estimated geometry, and visual memory. Its understanding of object state, physical causality, and long‑term task structure remains limited to their visible consequences.
• Conceived as a long‑term, open‑source project, AlayaWorld is intended to provide an extensible foundation for future research on interactive video world models and generative reality.