Summary (Overview)

• Hy-Nomas is an 8B-parameter interactive video world model built on a diffusion transformer (DiT) backbone, supporting free-form text instructions and continuous camera control across multiple interaction turns within a single unified model.
• The central design contribution is action-aware causal attention, which separates persistent descriptive context (accessible across segments) from temporally localized action instructions (restricted to their originating segment), reducing action-order errors and cross-segment action repetition.
• A vision-language Prompt Enhancer (PE) translates user requests and multimodal interaction histories into structured conditions for the next video segment, enabling multi-turn interaction without full-model fine-tuning.
• A multi-stage training curriculum progressively introduces causal generation, text interaction, camera control, and training on model-generated histories (segment forcing), followed by diffusion distillation for efficient few-step inference.
• In human evaluation across four benchmarks (multi-turn text, camera navigation, interleaved control, spatial memory), Hy-Nomas outperforms existing world models (Genie 3, LingBot-World v2, Happy Oyster) in instruction following and physical plausibility, while showing competitive spatial memory.

---

Introduction and Theoretical Foundation

Background and Motivation

Interactive world generation requires models to:

• Follow successive commands while preserving relevant history and action consequences
• Handle free-form text instructions and continuous camera control within the same session
• Generate variable-length segments conditioned on current inputs and cached history

The authors identify several key challenges:

1. Causality: A pretrained bidirectional DiT must be adapted to generate successive segments without information leakage, while supporting efficient reuse of cached context (KV cache).
2. Error compounding: Each generated segment becomes evidence for the next; small errors can compound into appearance drift and scene breakdown.
3. Action temporal alignment: Persistent scene information must remain accessible, while action instructions should take effect only during their intended time intervals.
4. Unified control: The model must coordinate free-form semantic commands with geometric camera controls.
5. Physical causality: Learning plausible action-to-state transitions requires training data that connects actions with observable consequences.

Theoretical Foundation

The model is built upon HunyuanVideo 1.5 (Wu et al., 2025) and uses:

• Flow matching objective (Lipman et al., 2023): Given clean latent $Z$ and Gaussian noise $\epsilon$, with $t \in [0,1]$:

$$Z_t = (1-t)Z + t\epsilon, \quad V_t = \epsilon - Z \tag{2}$$

• DCAE-V video autoencoder (Chen et al., 2025) with 32× spatial and 4× temporal compression:

$$Z = \mathcal{E}(X) \in \mathbb{R}^{B \times 32 \times F_z \times H_z \times W_z}, \quad F_z = \left\lfloor \frac{F-1}{4} \right\rfloor + 1, \quad H_z = \left\lfloor \frac{H}{32} \right\rfloor, \quad W_z = \left\lfloor \frac{W}{32} \right\rfloor \tag{1}$$

• Dual-stream MMDiT (Esser et al., 2024) with 54 blocks, hidden width 2048, and 16 attention heads (128 dimensions each)

---

Methodology

Data Pipeline

The data pipeline (Figure 2) includes:

1. Filtering: Videos are split into 5-minute segments, divided at shot boundaries (using fine-tuned OmniShotCut), transcoded, and filtered by duration, aspect ratio, resolution, motion, clarity, and aesthetic scores.

2. Deduplication: Visual embeddings extracted via Qwen3-VL-Embedding, hierarchical clustering, and SemDeDup for semantic deduplication.

3. Taxonomy labeling: A fine-tuned VLM labels videos according to a hierarchical taxonomy covering content categories (interaction types, object manipulation, motion) and visual attributes (camera perspective, subject type, recording methods).

4. Camera trajectories: ViPE (Huang et al., 2025a) estimates camera trajectories, supplemented by a trained inverse dynamics model (IDM) that predicts discrete camera motion from video. Four-step filtering rejects static scenes, excessive motion, lens distortion, and unstable trajectories.

5. Structured captioning: Captions use structured JSON with:

   • Video-level fields: perspective, style, shared subject list with identifiers
   • Segment-level fields: scene descriptions interleaved with camera-motion sentences (tagged with <CameraN>), and separately recorded subject actions with timestamps
   • Camera-motion sentences are removed when ViPE trajectories are available (replaced by explicit trajectory supervision); retained otherwise as textual supervision

Model Architecture

Segment-wise Causal Attention: Attention is bidirectional within each segment and causal across segments. Query tokens in segment $j$ attend to all tokens from segments $0, \ldots, j$, but not future segments.

Action-Aware Causal Attention (Figure 6b): Extends segment-wise causal attention with finer-grained masking:

• Text tokens divided into a descriptive head (environment description) and action tails $a_0, \ldots, a_n$ (individual actions with temporal windows)
• Query tokens within an action window attend only to associated action text and the descriptive head
• Descriptive heads remain accessible across segments; action tails are restricted to their originating segment

Camera-Trajectory Conditioning:

• Frame-to-frame relative motion encoded as $\Delta P_t = P_{t-1}^{-1}P_t$, with first increment set to identity
• Translations normalized by the 99th percentile of translation magnitudes
• Each group of four consecutive 4×4 transformation matrices flattened into a 64-dimensional vector
• Cross-attention injection: $\widetilde{V} = V + \text{CrossAttn}(V, C)$ (Equation 3), where visual tokens serve as queries and camera tokens as keys/values

Training Infrastructure

• Sequence parallelism: Ulysses-style SP with $N_{SP}=8$, supporting sequences up to 360K tokens
• FSDP2 hybrid sharding: Parameters, gradients, and optimizer states sharded across the 8-GPU SP group, replicated across DP groups
• FLOPs-based packing: Per-sample FLOPs estimated as $F_i = \alpha n_i + \beta A_i$ (Equation 6), where $A_i$ counts unmasked query-key pairs and $n_i$ is token count
• Flex-Flash-Attention: Structured masks represented as compact attention slices
• Distributed Muon optimizer for weight matrices, AdamW for embeddings and biases

Training Curriculum (Table 2)

Segment Forcing: The model processes segments sequentially, applies flow-matching MSE loss only to the current segment, and constructs context for the next segment via one-step denoising:

$$\tilde{\mathbf{x}}_{s, t_{\text{ctx}}} = \mathbf{x}_{s, t} + (t_{\text{ctx}} - t)\hat{\mathbf{v}}_s, \quad t_{\text{ctx}} \sim \mathcal{U}(0.1, 0.3) \tag{7}$$

Diffusion Distillation: Two-stage process:

• Stage I: Causal PCM initialization for few-step generation
• Stage II: DDMD + adversarial refinement under hybrid forcing, with 4 NFEs per segment

Inference

Prompt Enhancer (PE): A fine-tuned Qwen3.8-27B VLM that converts user input + history into structured captions:

$$y_t \sim p_\theta(\cdot | S, u_t, H_{t-1}, V_{<t}, m_t) \tag{8}$$

Optimization: INT8 KV-cache quantization, sliding-window attention, CPU offloading, sparse attention (25% of video blocks retained after first 2 denoising steps), and a distilled PixelUMM decoder for efficient single-GPU inference on RTX 4090.

---

Empirical Validation / Results

Hy-Nomas-Bench

Four benchmarks totaling 280 test cases:

Key Results

Bench-1 (Multi-turn text):

• Hy-Nomas wins 65.0% vs. HappyOyster-Directing and 75.0% vs. LingBot-World v2 on prompt following
• Wins 45.0% and 68.3% on physical plausibility

Bench-2 (Camera control): Hy-Nomas consistently achieves higher win rates than loss rates against Genie 3, HappyOyster-Wandering, and LingBot-World v2.

Bench-3 (Interleaved control): Hy-Nomas achieves 75.0% win rate on prompt following and 73.3% on physical plausibility vs. LingBot-World v2; camera control more comparable (26.7% win).

Bench-4 (Spatial memory):

• vs. Genie 3: 13.3% win, 56.7% tie, 30.0% loss (Genie 3 better)
• vs. HappyOyster: 30.0% win, 46.7% tie, 23.3% loss (comparable)
• vs. LingBot-World v2: 58.3% win, 1.7% loss (substantially better)

LoRA Adaptation Results

• Cinematic LoRA: Improves visual quality and prompt following (35% win vs. 24% loss on Bench-1 prompt following) but shows regression in camera control (19% win vs. 37% loss), likely due to limited camera trajectory diversity in SFT data
• Robotics LoRA: Enables out-of-distribution dual-arm task generation that the base model fails at

PE Results

• Hy-Nomas-PE substantially outperforms base Qwen3.8-27B on prompt following and physical plausibility, narrowing the gap with Kimi-K3
• Nearly 100% success rate in adhering to required response format (vs. ~50% for Qwen3.8-27B, ~70% for Kimi-K3)

---

Theoretical and Practical Implications

Theoretical Implications

1. Action-aware causal attention provides a principled mechanism for separating persistent context from temporally localized instructions, addressing fundamental challenges in autoregressive world generation.

2. The results support the hypothesis that detailed, structured captions reduce the burden on the diffusion transformer to "imagine" unspecified content, allowing it to focus on rendering described scenes and actions correctly.

3. The segment forcing approach demonstrates that training on model-generated histories (addressing the training-inference mismatch) is critical for stable multi-segment generation.

Practical Implications

1. Lightweight adaptation: The LoRA workflow enables downstream customization (cinematic style, robotics) with ~100 task-specific samples, without full-model fine-tuning or repeating distillation.

2. Deployment accessibility: Inference optimizations (INT8 quantization, SWA, CPU offloading, sparse attention) enable generation on a single NVIDIA RTX 4090 GPU (20.53 GiB peak for 100-second video at 1280×704).

3. Unified interaction: Supporting both text and camera control within one model eliminates the need for session resets or switching between specialized models.

4. Benchmark contribution: Hy-Nomas-Bench provides a reproducible evaluation framework covering four distinct world-model capabilities, released for community use.

---

Conclusion

Hy-Nomas demonstrates that an 8B-parameter diffusion transformer can effectively serve as an interactive world model supporting free-form text instructions, continuous camera navigation, and interleaved control. The key innovations—action-aware causal attention, structured chronological captions, and a progressive training curriculum—enable strong performance in instruction following and physical plausibility compared to larger or more specialized models.

Remaining challenges identified by the authors:

1. Spatial memory: Hy-Nomas trails Genie 3 in maintaining long-term spatial consistency. Future work could incorporate more looped camera trajectory data during training or explicit memory retrieval mechanisms.

2. Agentic memory management: Long-horizon generation requires offloading older context to CPU/storage and selectively retrieving relevant information. Different information types (geometry, reference images, text summaries) may benefit from different memory representations.

3. Out-of-view state updates: Retrieval alone cannot account for changes occurring outside the field of view (e.g., a wall clock advancing). Future agents could compute updated states from elapsed simulation time.

4. Gaming and robotics: Real-time interaction requires further latency reduction, while robotics requires predictions that match executable action outcomes (e.g., grasp success).

5. Scaling: Questions remain about whether larger models (e.g., MoE architectures) translate into better world-model capabilities beyond visual quality improvements.

The authors release model weights, code, and benchmarks to support reproducible research on interactive world generation.