Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

Summary (Overview)

• Core problem: Multimodal Large Language Model (MLLM) representations are often bypassed by Video Diffusion Transformers (DiTs) through native text-encoder and VAE-reference pathways, making them ineffective for cinematic video generation tasks that require temporal inference from sparse inputs and factor-aware control.

• Key contribution: The paper proposes Factor-Aware Predictive Learning (FAPL), a three-stage training framework that (1) grounds the DiT in MLLM latents as the sole content condition, (2) uses factor-span replacement between paired videos to supervise factor-aware changes and preservation, and (3) freezes the trained DiT while optimizing the MLLM through shared-target predictive learning.

• Results: FAPL consistently outperforms baselines (Omni-Video 2 at 1.3B and 14B, UniVideo at 13B) across three tasks: sparse-to-video generation (61% vs. 48% Motion Order), factor-aware editing (58% vs. 42% ∩-Acc), and multi-shot consistency (FaceSim-Arc 0.523 vs. 0.451).

• Key insight: MLLM feature bypass is a shared bottleneck for both temporal inference and factor-aware control; making representations "actionable" requires structured supervision that explicitly couples representation learning with generation objectives.

• Methodological innovation: FAPL introduces latent interventions (factor-span replacement) without assuming fully disentangled representations, and uses a fixed generator to direct MLLM learning toward generation-compatible representations.

---

Introduction and Theoretical Foundation

Background and Motivation

Cinematic video generation aims to help creators explore content, performances, and camera choices before physical production. Beyond visual realism, it requires two capabilities:

1. Temporal inference from limited inputs: Constructing plausible action sequences and camera motion from scripts and a few visual references, which only sparsely specify how a shot unfolds.
2. Factor-aware control: Character, scene, action, and camera must follow their respective instructions while remaining compatible (e.g., changing a character should preserve the requested action).

The MLLM Bypass Problem

The paper identifies a critical gap: making MLLM features available is different from making them effective for generation. Two obstacles remain:

• MLLM feature bypass: Native text-encoder and VAE-reference pathways can reduce the generation loss without substantial reliance on MLLM latents. The DiT may rely primarily on these familiar pathways, making MLLM representations effectively ignored.
• Insufficient task-structured supervision: Standard video-level training does not explicitly associate interventions on specific MLLM spans with intended output changes, nor does it train the MLLM to produce temporally informative representations from sparse inputs.

Theoretical Foundation

The framework builds on:

• Flow matching (Lipman et al., 2022): The DiT is trained to predict velocity $v^* = \partial z_t / \partial t$ from noisy latent $z_t$.
• MLLM-conditioned video diffusion (Yang et al., 2026a; Wei et al., 2026): Connecting MLLMs to pretrained Video DiTs via a connector that maps hidden states to the DiT conditioning space.
• Predictive representation learning (Assran et al., 2023): Joint-embedding predictive architectures learn representations by predicting target features from partial observations.

The central premise is that both the generator must learn how to use MLLM representations and the MLLM must learn what representations to produce for generation—a bidirectional adaptation problem.

---

Methodology

Framework: MLLM-Conditioned Video Diffusion

The generator connects an MLLM $F_\theta$ to a pretrained Video DiT $G_\phi$ via a connector $P_\eta$. The MLLM condition is:

$$C^{\mathrm{mllm}} = Z_{\theta,\eta}(u) = P_\eta\bigl(H_\theta(u)\bigr) \tag{1}$$

where $u$ is a text-only or multimodal input. The DiT receives three concatenated conditions $C = [C^{\mathrm{text}}; C^{\mathrm{mllm}}; C^{\mathrm{ref}}]$, though FAPL uses only the MLLM latents as content conditions.

The flow matching objective:

$$\mathcal{L}_{\mathrm{FM}}(x \mid C; \phi) = \mathbb{E}_{t,\epsilon} \left[\left\| G_\phi(z_t, t \mid C) - v^* \right\|_2^2\right] \tag{2}$$

Paired Data Construction

Two Observation Modes, One Video Target: For each video $x$ with description $p$, two inputs are constructed: $u^t = p$ (text-only) and $u^{tv} = (p, x)$ (text-plus-complete-video), both supervised with the same target video $x$.

Selective Factor Changes: The editing set $\mathcal{D}_{\mathrm{edit}}$ contains tuples $(x^a, p^a, x^b, p^b, k)$ where only factor $k \in \mathcal{F} = \{\text{character, scene, action, camera}\}$ is modified. Descriptions are structured prompts with four ordered fields, enabling token-span localization for intervention.

Stage I: MLLM-Only Generator Grounding

Freeze MLLM ($\theta_0$) and connector ($\eta_0$), train only the DiT with MLLM latents as the sole content condition:

$$\mathcal{L}_{\mathrm{ground}}(\phi) = \mathbb{E}_{(x,p) \sim \mathcal{D}_{\mathrm{obs}}} \left[\sum_{m \in \{\mathrm{t}, \mathrm{tv}\}} \rho_m \mathcal{L}_{\mathrm{FM}}\left(x \mid Z_{\theta_0,\eta_0}(u^m); \phi\right)\right] \tag{3}$$

with $\rho_t + \rho_{tv} = 1$. This isolates the MLLM pathway, forcing the DiT to learn how to use its representations.

Stage II: Factor-Aware Control Improvement

Keep MLLM and connector frozen. Factor-span replacement swaps the factor-$k$ span between paired examples:

$$Z_k^{a \leftarrow b} = \mathcal{R}_k(Z^a, Z^b), \qquad Z_k^{b \leftarrow a} = \mathcal{R}_k(Z^b, Z^a) \tag{4}$$

The editing objective:

$$\mathcal{L}_{\mathrm{edit}}(\phi) = \mathbb{E}_{(x^a,p^a,x^b,p^b,k)} \left[ \mathcal{L}_{\mathrm{FM}}\big(x^b \mid Z_k^{a \leftarrow b}; \phi\big) + \mathcal{L}_{\mathrm{FM}}\big(x^a \mid Z_k^{b \leftarrow a}; \phi\big) \right] \tag{5}$$

This supervises the DiT to realize target-factor changes while preserving non-targets, without requiring fully disentangled representations.

Stage III: Prediction from Limited Inputs

Freeze the trained DiT ($\phi^*$), jointly optimize the MLLM and connector:

$$\mathcal{L}_{\mathrm{obs}}(\theta, \eta) = \mathbb{E}_{(x,p) \sim \mathcal{D}_{\mathrm{obs}}} \left[\sum_{m \in \{\mathrm{t}, \mathrm{tv}\}} \pi_m \mathcal{L}_{\mathrm{FM}}\left(x \mid Z_{\theta,\eta}(u^m); \phi^*\right)\right] \tag{6}$$

The fixed DiT directs MLLM learning toward generation-compatible representations. Text-only mode trains prediction from limited evidence; complete-video mode provides complementary reconstruction supervision.

Inference

Three applications in one architecture: (i) generation from scripts/references, (ii) factor-aware control via structured prompt editing and span replacement, (iii) multi-shot generation with shared references.

---

Empirical Validation / Results

Implementation Details

• Built on UniVideo (13B DiT); compared with UniVideo and Omni-Video 2 (1.3B, 14B DiTs)
• Stages I–III trained for 10K, 15K, and 10K steps (AdamW, LR $1 \times 10^{-5}$, BF16)
• 100K quality-filtered source–target video pairs (25K per factor from CoinVE-200K and OmniVBench)
• Evaluation: CineFactor-10K (10K single-shot cases), ShotVerse-Bench (100 multi-shot sequences)

Main Results (Table 1)

Ablation Study (Table 2)

Key Findings

• Stage I improves action ordering (48→54%) and camera errors relative to pretrained model.
• Stage II: Factor swapping yields 57% vs. 48% ∩-Acc (no swapping control), showing benefit is concentrated in factor-aware editing rather than general generation.
• Stage III: Mixed-observation training improves Motion Order (57→61%) and camera errors more than text-only training, while retaining editing performance (57→58%).

Qualitative Results

• Sparse-to-video: Baselines miss specified details (e.g., long hair generated as short hair, missing hand gestures, static camera instead of requested orbit); FAPL follows specifications more closely.
• Factor-aware editing: Baselines alter non-target factors; FAPL confines changes to the selected factor while preserving others.
• Multi-shot: FAPL maintains shared character/scene appearance while varying action and camera across shots.

---

Theoretical and Practical Implications

Theoretical Implications

1. MLLM latent bypass is a fundamental bottleneck: The paper demonstrates that simply adding MLLM features to a conditioning interface does not guarantee their effective use. The DiT will preferentially rely on familiar pathways (text encoders, VAE references) unless explicitly trained to use MLLM latents.

2. Bidirectional adaptation is necessary: Both the generator (learning to decode MLLM representations) and the MLLM (learning to produce generation-compatible representations) must be adapted, with the other fixed. This establishes a principled training order: generator grounding → factor-aware supervision → MLLM prediction learning.

3. Latent interventions without disentanglement: Factor-span replacement works without requiring fully disentangled representations, showing that structured supervision can compensate for imperfect representation separability.

4. Fixed-generator predictive learning: Using a fixed generator to supervise MLLM learning avoids the competing objectives problem where simultaneous decoder adaptation defeats representation learning.

Practical Implications

1. Unified architecture for multiple tasks: FAPL achieves sparse-to-video generation, factor-aware editing, and multi-shot generation within a single architecture with no architectural changes.

2. Structured prompt engineering: The four-field prompt structure (character, scene, action, camera) provides a practical interface for cinematic control.

3. Compatibility with existing models: Stage I's latent-only grounding generalizes across MLLM–DiT models (demonstrated on Omni-Video 2 and UniVideo).

4. Data efficiency: The approach requires paired video data with factor-level annotations, which can be constructed from existing datasets (CoinVE-200K, OmniVBench).

---

Conclusion

Main takeaways:

• FAPL addresses the critical gap between MLLM understanding and its effective use in cinematic generation by tackling the MLLM feature bypass problem as a shared prerequisite.
• The three-stage training framework targets the right bottlenecks: generator grounding (Stage I), factor-aware control (Stage II), and limited-input prediction (Stage III), delivering substantial gains without architectural complexity.
• The paper demonstrates that reasoning guides controllable synthesis, while generation provides grounded supervision for learning to infer missing content from limited observations.

Future directions:

• Current MLLMs struggle with 3D blockout interpretation and complex camera reasoning for DiT control.
• The pretrained DiT limits visual fidelity and complex factor realization (e.g., realistic acting).
• Future work may incorporate stronger MLLMs and more capable video generators to push the boundaries of cinematic video generation.