Summary (Overview)

• Real-time generative rendering: AlayaRenderer-Flash accelerates the original AlayaRenderer from 0.56 FPS to 31.54 FPS, enabling interactive, prompt-controllable world rendering at playback rate.
• Autoregressive streaming: Reformulates the fixed-window bidirectional renderer into an autoregressive model that can process unbounded G-buffer streams from live game engines.
• Few-step distillation: Reduces the 50-step denoising schedule to 4 steps via a three-stage pipeline (guidance distillation, progressive step reduction, Mean Flow Distillation with GAN heads), preserving quality while cutting inference cost.
• Lightweight codecs: Replaces the expensive Wan VAE encoder/decoder with distilled tiny encoder and temporal decoder, reducing peak GPU memory from 22.6 GB to 16.2 GB and further boosting throughput.
• End-to-end interactive deployment: Integrated with a physics engine (SuperTuxKart), the system sustains 30 FPS during live gameplay, demonstrating the feasibility of real-time generative world modeling.

Introduction and Theoretical Foundation

The paper addresses the challenge of real-time generative world rendering. Traditional generative models that synthesize frames from text or control hints often fail to preserve scene structure and gameplay logic. A more principled approach is to keep the physics engine responsible for world dynamics and delegate appearance generation to a generative renderer conditioned on G-buffers—structured world states (depth, normals, albedo, roughness, metallic) exported by game engines. This engine-plus-generative-renderer paradigm preserves geometry, physics, and interactivity while allowing text-driven restyling.

AlayaRenderer (Huang et al., 2026b) is the state-of-the-art in this direction, but it is an offline renderer: its 50-step denoising process, expensive VAE encoding/decoding, and bidirectional fixed-window formulation prevent real-time deployment. AlayaRenderer-Flash overcomes these limitations through three complementary improvements: autoregressive streaming, few-step distillation, and lightweight codecs, enabling interactive, prompt-controllable rendering alongside live gameplay.

Methodology

G-Buffer-Conditioned Latent Rendering

The base renderer (AlayaRenderer) is built on Wan 2.1, a latent video diffusion model. The game engine provides five G-buffer channels (albedo, depth, metallic, normal, roughness), each encoded by the Wan causal 3D VAE. The G-buffer condition $g_k$ is concatenated with the noisy RGB latent $x_k$ before the first 3D patch embedding:

$$h_k = \text{PatchEmbed}([x_k, g_k])$$

where $h_k$ is the input to the diffusion transformer. This minimal modification widens the input projection to accommodate condition channels.

Autoregressive Streaming

AlayaRenderer-Flash operates autoregressively at the chunk level (4 latent frames per chunk). Previously generated chunks are retained as history in a three-tier compression hierarchy: recent history at full fidelity, distant history progressively compressed. The first generated latent frame is prepended as a global appearance anchor. During denoising, all retained history is inserted into self-attention alongside the current chunk. A text sink (persistent key–value entries from the style-prompt embedding) is added to every self-attention layer, enabling prompt controllability over arbitrary-length rollouts. The VAE encoder/decoder are also made causal with persistent temporal caches.

Few-Step Distillation

A three-stage pipeline reduces denoising from 50 to 4 steps:

1. Guidance Distillation: Distills the teacher's CFG-combined velocity field at the target guidance scale into the student, enabling single-pass evaluation.
2. Progressive Step Reduction: Sequentially trains the student with 32, 16, 8, and 4 steps (Progressive Distillation) to provide stable initialization.
3. Mean Flow Distillation (MFD) under self-rollout: Refines the 4-step student using flow-based matching under autoregressive self-rollout (Self Forcing). Lightweight GAN heads are attached to intermediate features and final latent prediction (MFD+GAN) to recover high-frequency details.

Distilled Tiny Codecs

• Tiny decoder: Adopts the TAEHV architecture, initialized from Wan 2.1 pretrained weights, then distilled from the frozen Wan VAE decoder using pixel reconstruction and perceptual loss.
• Tiny G-buffer encoder: A shared network that predicts the G-buffer condition $g_k$ in a single forward pass (instead of five separate Wan VAE passes). First distilled to match Wan VAE encoder latents, then jointly fine-tuned with the renderer.

Real-Time Deployment

All components (tiny encoder, 4-step diffusion, tiny decoder) operate autoregressively with persistent temporal states across chunk boundaries, enabling continuous rendering over unbounded streams. The pipeline is executed as a single optimized streaming system.

Empirical Validation / Results

Experimental Setup

• Dataset: Engine-captured Black Myth: Wukong at 1280×720, 30 FPS. 1,352 training clips, 131 test clips (150 frames each, 5 seconds). Model input resolution: 832×448.
• Metrics: Content preservation (CLIP image-image similarity $S_{\text{CLIP-I}}$), temporal stability (warped temporal LPIPS $\text{tLPIPS}_{\text{warp}}$), cross-window consistency (Boundary MSE, Boundary SSIM), prompt controllability (contrastive CLIP margin $M_{\text{CLIP}}$), and deployment performance (FPS, VRAM).

Progressive Design Analysis (Table 1)

Key observations: Autoregressive streaming improves content similarity and boundary consistency but reduces temporal stability. 4-step distillation recovers temporal stability and boosts FPS to 6.30. Lightweight codecs further increase FPS to 31.54 and reduce VRAM to 16.2 GB while maintaining or improving quality.

Capability and Performance Comparison (Tables 2 & 3)

Table 2: Capability comparison

Table 3: Quantitative comparison (5-second protocol)

AlayaRenderer-Flash achieves the best trade-off: highest content similarity, lowest FVD, best temporal stability, and real-time FPS. DiffusionRenderer (bidirectional) achieves higher $S_{\text{CLIP-I}}$ (0.870) and lower FVD (335.5) under a favorable offline protocol, but runs at only 1.10 FPS.

Qualitative Results

Figure 3 shows that AlayaRenderer-Flash maintains stable illumination, scene geometry, and camera motion over long rollouts, outperforming RGB ↔ X (temporal flicker) and FrameDiffuser (appearance drift).

Prompt Switching on Long Rollouts

On 637-frame sequences with 8 prompt switches, AlayaRenderer-Flash follows transitions smoothly without ghosting or boundary artifacts, preserving scene structure and motion.

End-to-End Interactive Deployment

Integrated with SuperTuxKart, the system runs at 31.54 FPS on a single NVIDIA H200 GPU, sustaining 30 FPS during live gameplay including engine-side overhead.

Theoretical and Practical Implications

• Theoretical: Demonstrates that a generative renderer can replace the shading stage of a real-time game engine while preserving world dynamics. The autoregressive streaming formulation with hierarchical history compression and text sinks provides a principled way to maintain long-horizon consistency and prompt controllability.
• Practical: Enables interactive, user-controllable visual restyling of games without modifying geometry, materials, or gameplay logic. The 31.54 FPS throughput makes generative rendering viable for real-time applications beyond games (e.g., simulation, VR, live streaming). The distilled codecs and few-step diffusion reduce hardware requirements, making deployment on consumer GPUs plausible.
• Limitations: Aggressive step reduction suppresses high-frequency details (partially recovered by GAN heads). The method is currently demonstrated on a specific game dataset; generalization to diverse engines and scenes requires further study.

Conclusion

AlayaRenderer-Flash transforms AlayaRenderer from an offline diffusion renderer into a real-time streaming generative renderer. By combining autoregressive streaming, progressive four-step distillation, and lightweight codecs, it achieves 31.54 FPS while preserving rendering quality, prompt controllability, temporal consistency, and structural fidelity. Integrated with a physics engine, it sustains 30 FPS during live gameplay, demonstrating the practical feasibility of streaming generative world rendering. This work represents a step toward real-time generative world models and AI-native interactive environments.