# Hy-Nomas: Towards Free-form Interactive World Generation > Hy-Nomas is an 8B interactive video world model that unifies free-form text and camera control via action-aware causal attention, outperforming larger baselines in instruction-following. - **Source:** [arXiv](https://arxiv.org/abs/2610.36729) - **Published:** 2026-10-29 - **Permalink:** https://picx.dev/p/8H5X8y - **Whiteboard:** https://picx.dev/p/8H5X8y/image ## Summary ## Summary (Overview) - **Hy-Nomas** is an 8B-parameter interactive video world model built on a diffusion transformer (DiT) backbone, supporting **free-form text instructions** and **continuous camera control** across multiple interaction turns within a single unified model. - The central design contribution is **action-aware causal attention**, which separates persistent descriptive context (accessible across segments) from temporally localized action instructions (restricted to their originating segment), reducing action-order errors and cross-segment action repetition. - A **vision-language Prompt Enhancer (PE)** translates user requests and multimodal interaction histories into structured conditions for the next video segment, enabling multi-turn interaction without full-model fine-tuning. - A **multi-stage training curriculum** progressively introduces causal generation, text interaction, camera control, and training on model-generated histories (segment forcing), followed by diffusion distillation for efficient few-step inference. - In human evaluation across four benchmarks (multi-turn text, camera navigation, interleaved control, spatial memory), **Hy-Nomas outperforms existing world models** (Genie 3, LingBot-World v2, Happy Oyster) in instruction following and physical plausibility, while showing competitive spatial memory. --- ## Introduction and Theoretical Foundation ### Background and Motivation Interactive world generation requires models to: - Follow **successive commands** while preserving relevant history and action consequences - Handle **free-form text instructions** and **continuous camera control** within the same session - Generate **variable-length segments** conditioned on current inputs and cached history The authors identify several key challenges: 1. **Causality**: A pretrained bidirectional DiT must be adapted to generate successive segments without information leakage, while supporting efficient reuse of cached context (KV cache). 2. **Error compounding**: Each generated segment becomes evidence for the next; small errors can compound into appearance drift and scene breakdown. 3. **Action temporal alignment**: Persistent scene information must remain accessible, while action instructions should take effect only during their intended time intervals. 4. **Unified control**: The model must coordinate free-form semantic commands with geometric camera controls. 5. **Physical causality**: Learning plausible action-to-state transitions requires training data that connects actions with observable consequences. ### Theoretical Foundation The model is built upon **HunyuanVideo 1.5** (Wu et al., 2025) and uses: - **Flow matching objective** (Lipman et al., 2023): Given clean latent $Z$ and Gaussian noise $\epsilon$, with $t \in [0,1]$: $$Z_t = (1-t)Z + t\epsilon, \quad V_t = \epsilon - Z \tag{2}$$ - **DCAE-V video autoencoder** (Chen et al., 2025) with 32× spatial and 4× temporal compression: $$Z = \mathcal{E}(X) \in \mathbb{R}^{B \times 32 \times F_z \times H_z \times W_z}, \quad F_z = \left\lfloor \frac{F-1}{4} \right\rfloor + 1, \quad H_z = \left\lfloor \frac{H}{32} \right\rfloor, \quad W_z = \left\lfloor \frac{W}{32} \right\rfloor \tag{1}$$ - **Dual-stream MMDiT** (Esser et al., 2024) with 54 blocks, hidden width 2048, and 16 attention heads (128 dimensions each) --- ## Methodology ### Data Pipeline The data pipeline (Figure 2) includes: 1. **Filtering**: Videos are split into 5-minute segments, divided at shot boundaries (using fine-tuned OmniShotCut), transcoded, and filtered by duration, aspect ratio, resolution, motion, clarity, and aesthetic scores. 2. **Deduplication**: Visual embeddings extracted via Qwen3-VL-Embedding, hierarchical clustering, and SemDeDup for semantic deduplication. 3. **Taxonomy labeling**: A fine-tuned VLM labels videos according to a hierarchical taxonomy covering content categories (interaction types, object manipulation, motion) and visual attributes (camera perspective, subject type, recording methods). 4. **Camera trajectories**: ViPE (Huang et al., 2025a) estimates camera trajectories, supplemented by a trained inverse dynamics model (IDM) that predicts discrete camera motion from video. Four-step filtering rejects static scenes, excessive motion, lens distortion, and unstable trajectories. 5. **Structured captioning**: Captions use structured JSON with: - **Video-level fields**: perspective, style, shared subject list with identifiers - **Segment-level fields**: scene descriptions interleaved with camera-motion sentences (tagged with ``), and separately recorded subject actions with timestamps - Camera-motion sentences are removed when ViPE trajectories are available (replaced by explicit trajectory supervision); retained otherwise as textual supervision ### Model Architecture **Segment-wise Causal Attention**: Attention is bidirectional within each segment and causal across segments. Query tokens in segment $j$ attend to all tokens from segments $0, \ldots, j$, but not future segments. **Action-Aware Causal Attention** (Figure 6b): Extends segment-wise causal attention with finer-grained masking: - Text tokens divided into a **descriptive head** (environment description) and **action tails** $a_0, \ldots, a_n$ (individual actions with temporal windows) - Query tokens within an action window attend only to associated action text and the descriptive head - Descriptive heads remain accessible across segments; action tails are restricted to their originating segment **Camera-Trajectory Conditioning**: - Frame-to-frame relative motion encoded as $\Delta P_t = P_{t-1}^{-1}P_t$, with first increment set to identity - Translations normalized by the 99th percentile of translation magnitudes - Each group of four consecutive 4×4 transformation matrices flattened into a 64-dimensional vector - Cross-attention injection: $\widetilde{V} = V + \text{CrossAttn}(V, C)$ (Equation 3), where visual tokens serve as queries and camera tokens as keys/values ### Training Infrastructure - **Sequence parallelism**: Ulysses-style SP with $N_{SP}=8$, supporting sequences up to 360K tokens - **FSDP2 hybrid sharding**: Parameters, gradients, and optimizer states sharded across the 8-GPU SP group, replicated across DP groups - **FLOPs-based packing**: Per-sample FLOPs estimated as $F_i = \alpha n_i + \beta A_i$ (Equation 6), where $A_i$ counts unmasked query-key pairs and $n_i$ is token count - **Flex-Flash-Attention**: Structured masks represented as compact attention slices - **Distributed Muon optimizer** for weight matrices, AdamW for embeddings and biases ### Training Curriculum (Table 2) | Stage | Training Regime | Primary Data/Supervision | Main Objective | |-------|----------------|--------------------------|----------------| | 1 | Latent-space adaptation | HunyuanVideo 1.5 init; 20K steps (I/O layers) + 60K steps (full) | Transfer to DCAE-V latent space | | 2a | Segment-causal CPT | Broad video corpus with segment captions | Establish segment-wise causal generation | | 2b | Action-aware CPT | Curated videos with localized action captions | Learn action timing; expand context to 360K tokens | | 2c | Camera-conditioned CPT | Videos with/without trajectories (1:1) | Add continuous camera control | | 3 | High-quality annealing | High-quality videos, Kimi K3 captions | Refine visual quality and control | | 4 | Segment forcing | Subset of annealing data | Align training with autoregressive inference | | 5 | Diffusion distillation | Causal PCM + DDMD + adversarial | Few-step generation | **Segment Forcing**: The model processes segments sequentially, applies flow-matching MSE loss only to the current segment, and constructs context for the next segment via one-step denoising: $$\tilde{\mathbf{x}}_{s, t_{\text{ctx}}} = \mathbf{x}_{s, t} + (t_{\text{ctx}} - t)\hat{\mathbf{v}}_s, \quad t_{\text{ctx}} \sim \mathcal{U}(0.1, 0.3) \tag{7}$$ **Diffusion Distillation**: Two-stage process: - **Stage I**: Causal PCM initialization for few-step generation - **Stage II**: DDMD + adversarial refinement under hybrid forcing, with 4 NFEs per segment ### Inference **Prompt Enhancer (PE)**: A fine-tuned Qwen3.8-27B VLM that converts user input + history into structured captions: $$y_t \sim p_\theta(\cdot | S, u_t, H_{t-1}, V_{