# Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

> Factor-aware predictive learning, which grounds video diffusion in MLLM latents via factor-span replacement and fixed-generator supervision, boosts cinematic generation accuracy by 13% over baselines.

- **Source:** [arXiv](https://arxiv.org/abs/2610.30926)
- **Published:** 2026-10-24
- **Permalink:** https://picx.dev/p/Ob5HCZ
- **Whiteboard:** https://picx.dev/p/Ob5HCZ/image

## Summary

# Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

## Summary (Overview)

- **Core problem:** Multimodal Large Language Model (MLLM) representations are often bypassed by Video Diffusion Transformers (DiTs) through native text-encoder and VAE-reference pathways, making them ineffective for cinematic video generation tasks that require temporal inference from sparse inputs and factor-aware control.

- **Key contribution:** The paper proposes **Factor-Aware Predictive Learning (FAPL)**, a three-stage training framework that (1) grounds the DiT in MLLM latents as the sole content condition, (2) uses factor-span replacement between paired videos to supervise factor-aware changes and preservation, and (3) freezes the trained DiT while optimizing the MLLM through shared-target predictive learning.

- **Results:** FAPL consistently outperforms baselines (Omni-Video 2 at 1.3B and 14B, UniVideo at 13B) across three tasks: sparse-to-video generation (61% vs. 48% Motion Order), factor-aware editing (58% vs. 42% ∩-Acc), and multi-shot consistency (FaceSim-Arc 0.523 vs. 0.451).

- **Key insight:** MLLM feature bypass is a shared bottleneck for both temporal inference and factor-aware control; making representations "actionable" requires structured supervision that explicitly couples representation learning with generation objectives.

- **Methodological innovation:** FAPL introduces latent interventions (factor-span replacement) without assuming fully disentangled representations, and uses a fixed generator to direct MLLM learning toward generation-compatible representations.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Cinematic video generation aims to help creators explore content, performances, and camera choices before physical production. Beyond visual realism, it requires two capabilities:

1. **Temporal inference from limited inputs**: Constructing plausible action sequences and camera motion from scripts and a few visual references, which only sparsely specify how a shot unfolds.
2. **Factor-aware control**: Character, scene, action, and camera must follow their respective instructions while remaining compatible (e.g., changing a character should preserve the requested action).

### The MLLM Bypass Problem

The paper identifies a critical gap: **making MLLM features available is different from making them effective for generation**. Two obstacles remain:

- **MLLM feature bypass**: Native text-encoder and VAE-reference pathways can reduce the generation loss without substantial reliance on MLLM latents. The DiT may rely primarily on these familiar pathways, making MLLM representations effectively ignored.
- **Insufficient task-structured supervision**: Standard video-level training does not explicitly associate interventions on specific MLLM spans with intended output changes, nor does it train the MLLM to produce temporally informative representations from sparse inputs.

### Theoretical Foundation

The framework builds on:

- **Flow matching** (Lipman et al., 2022): The DiT is trained to predict velocity $v^* = \partial z_t / \partial t$ from noisy latent $z_t$.
- **MLLM-conditioned video diffusion** (Yang et al., 2026a; Wei et al., 2026): Connecting MLLMs to pretrained Video DiTs via a connector that maps hidden states to the DiT conditioning space.
- **Predictive representation learning** (Assran et al., 2023): Joint-embedding predictive architectures learn representations by predicting target features from partial observations.

The central premise is that both the **generator must learn how to use MLLM representations** and the **MLLM must learn what representations to produce for generation**—a bidirectional adaptation problem.

---

## Methodology

### Framework: MLLM-Conditioned Video Diffusion

The generator connects an MLLM $F_\theta$ to a pretrained Video DiT $G_\phi$ via a connector $P_\eta$. The MLLM condition is:

$$C^{\mathrm{mllm}} = Z_{\theta,\eta}(u) = P_\eta\bigl(H_\theta(u)\bigr) \tag{1}$$

where $u$ is a text-only or multimodal input. The DiT receives three concatenated conditions $C = [C^{\mathrm{text}}; C^{\mathrm{mllm}}; C^{\mathrm{ref}}]$, though FAPL uses only the MLLM latents as content conditions.

The flow matching objective:

$$\mathcal{L}_{\mathrm{FM}}(x \mid C; \phi) = \mathbb{E}_{t,\epsilon} \left[\left\| G_\phi(z_t, t \mid C) - v^* \right\|_2^2\right] \tag{2}$$

### Paired Data Construction

**Two Observation Modes, One Video Target:** For each video $x$ with description $p$, two inputs are constructed: $u^t = p$ (text-only) and $u^{tv} = (p, x)$ (text-plus-complete-video), both supervised with the same target video $x$.

**Selective Factor Changes:** The editing set $\mathcal{D}_{\mathrm{edit}}$ contains tuples $(x^a, p^a, x^b, p^b, k)$ where only factor $k \in \mathcal{F} = \{\text{character, scene, action, camera}\}$ is modified. Descriptions are structured prompts with four ordered fields, enabling token-span localization for intervention.

### Stage I: MLLM-Only Generator Grounding

Freeze MLLM ($\theta_0$) and connector ($\eta_0$), train only the DiT with MLLM latents as the **sole** content condition:

$$\mathcal{L}_{\mathrm{ground}}(\phi) = \mathbb{E}_{(x,p) \sim \mathcal{D}_{\mathrm{obs}}} \left[\sum_{m \in \{\mathrm{t}, \mathrm{tv}\}} \rho_m \mathcal{L}_{\mathrm{FM}}\left(x \mid Z_{\theta_0,\eta_0}(u^m); \phi\right)\right] \tag{3}$$

with $\rho_t + \rho_{tv} = 1$. This isolates the MLLM pathway, forcing the DiT to learn how to use its representations.

### Stage II: Factor-Aware Control Improvement

Keep MLLM and connector frozen. **Factor-span replacement** swaps the factor-$k$ span between paired examples:

$$Z_k^{a \leftarrow b} = \mathcal{R}_k(Z^a, Z^b), \qquad Z_k^{b \leftarrow a} = \mathcal{R}_k(Z^b, Z^a) \tag{4}$$

The editing objective:

$$\mathcal{L}_{\mathrm{edit}}(\phi) = \mathbb{E}_{(x^a,p^a,x^b,p^b,k)} \left[ \mathcal{L}_{\mathrm{FM}}\big(x^b \mid Z_k^{a \leftarrow b}; \phi\big) + \mathcal{L}_{\mathrm{FM}}\big(x^a \mid Z_k^{b \leftarrow a}; \phi\big) \right] \tag{5}$$

This supervises the DiT to realize target-factor changes while preserving non-targets, without requiring fully disentangled representations.

### Stage III: Prediction from Limited Inputs

Freeze the trained DiT ($\phi^*$), jointly optimize the MLLM and connector:

$$\mathcal{L}_{\mathrm{obs}}(\theta, \eta) = \mathbb{E}_{(x,p) \sim \mathcal{D}_{\mathrm{obs}}} \left[\sum_{m \in \{\mathrm{t}, \mathrm{tv}\}} \pi_m \mathcal{L}_{\mathrm{FM}}\left(x \mid Z_{\theta,\eta}(u^m); \phi^*\right)\right] \tag{6}$$

The fixed DiT directs MLLM learning toward generation-compatible representations. Text-only mode trains prediction from limited evidence; complete-video mode provides complementary reconstruction supervision.

### Inference

Three applications in one architecture: (i) generation from scripts/references, (ii) factor-aware control via structured prompt editing and span replacement, (iii) multi-shot generation with shared references.

---

## Empirical Validation / Results

### Implementation Details

- Built on **UniVideo** (13B DiT); compared with UniVideo and Omni-Video 2 (1.3B, 14B DiTs)
- Stages I–III trained for 10K, 15K, and 10K steps (AdamW, LR $1 \times 10^{-5}$, BF16)
- 100K quality-filtered source–target video pairs (25K per factor from CoinVE-200K and OmniVBench)
- Evaluation: CineFactor-10K (10K single-shot cases), ShotVerse-Bench (100 multi-shot sequences)

### Main Results (Table 1)

| Method | Motion Order (%)↑ | Trans. Error↓ | Rot. Error (rad)↓ | ∩-Acc (%)↑ | Subject↑ | Scene↑ | FaceSim-Arc↑ |
|---|---|---|---|---|---|---|---|
| Omni-Video 2 (1.3B) | 33 | 0.1084 | 1.87 | 28 | 0.435 | 0.373 | 0.412 |
| Omni-Video 2 (14B) | 43 | 0.0817 | 1.49 | 37 | 0.476 | 0.423 | 0.468 |
| UniVideo (13B) | 48 | 0.0763 | 1.54 | 42 | 0.462 | 0.427 | 0.451 |
| **FAPL (Ours)** | **61** | **0.0586** | **1.02** | **58** | **0.516** | **0.471** | **0.523** |

### Ablation Study (Table 2)

| Exp | Training Configuration | Motion Order (%)↑ | Trans. Error↓ | Rot. Error (rad)↓ | ∩-Acc (%)↑ |
|---|---|---|---|---|---|
| 0 | Pretrained UniVideo | 48 | 0.0763 | 1.54 | 42 |
| 1 | Stage I only | 54 | 0.0708 | 1.38 | 44 |
| 2 | Stage II (no factor swapping) | 54 | 0.0654 | 1.30 | 48 |
| 3 | Stage II (with factor swapping) | 53 | 0.0671 | 1.32 | **57** |
| 4 | Stage III (text-only) | 57 | 0.0629 | 1.26 | 56 |
| 5 | **Stage III (Full FAPL)** | **61** | **0.0586** | **1.02** | **58** |

### Key Findings

- **Stage I** improves action ordering (48→54%) and camera errors relative to pretrained model.
- **Stage II**: Factor swapping yields 57% vs. 48% ∩-Acc (no swapping control), showing benefit is concentrated in factor-aware editing rather than general generation.
- **Stage III**: Mixed-observation training improves Motion Order (57→61%) and camera errors more than text-only training, while retaining editing performance (57→58%).

### Qualitative Results

- **Sparse-to-video**: Baselines miss specified details (e.g., long hair generated as short hair, missing hand gestures, static camera instead of requested orbit); FAPL follows specifications more closely.
- **Factor-aware editing**: Baselines alter non-target factors; FAPL confines changes to the selected factor while preserving others.
- **Multi-shot**: FAPL maintains shared character/scene appearance while varying action and camera across shots.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **MLLM latent bypass is a fundamental bottleneck**: The paper demonstrates that simply adding MLLM features to a conditioning interface does not guarantee their effective use. The DiT will preferentially rely on familiar pathways (text encoders, VAE references) unless explicitly trained to use MLLM latents.

2. **Bidirectional adaptation is necessary**: Both the generator (learning to decode MLLM representations) and the MLLM (learning to produce generation-compatible representations) must be adapted, with the other fixed. This establishes a principled training order: generator grounding → factor-aware supervision → MLLM prediction learning.

3. **Latent interventions without disentanglement**: Factor-span replacement works without requiring fully disentangled representations, showing that structured supervision can compensate for imperfect representation separability.

4. **Fixed-generator predictive learning**: Using a fixed generator to supervise MLLM learning avoids the competing objectives problem where simultaneous decoder adaptation defeats representation learning.

### Practical Implications

1. **Unified architecture for multiple tasks**: FAPL achieves sparse-to-video generation, factor-aware editing, and multi-shot generation within a single architecture with no architectural changes.

2. **Structured prompt engineering**: The four-field prompt structure (character, scene, action, camera) provides a practical interface for cinematic control.

3. **Compatibility with existing models**: Stage I's latent-only grounding generalizes across MLLM–DiT models (demonstrated on Omni-Video 2 and UniVideo).

4. **Data efficiency**: The approach requires paired video data with factor-level annotations, which can be constructed from existing datasets (CoinVE-200K, OmniVBench).

---

## Conclusion

**Main takeaways:**
- FAPL addresses the critical gap between MLLM understanding and its effective use in cinematic generation by tackling the MLLM feature bypass problem as a shared prerequisite.
- The three-stage training framework targets the right bottlenecks: generator grounding (Stage I), factor-aware control (Stage II), and limited-input prediction (Stage III), delivering substantial gains without architectural complexity.
- The paper demonstrates that **reasoning guides controllable synthesis, while generation provides grounded supervision for learning to infer missing content from limited observations**.

**Future directions:**
- Current MLLMs struggle with 3D blockout interpretation and complex camera reasoning for DiT control.
- The pretrained DiT limits visual fidelity and complex factor realization (e.g., realistic acting).
- Future work may incorporate stronger MLLMs and more capable video generators to push the boundaries of cinematic video generation.

---

_Markdown view of https://picx.dev/p/Ob5HCZ, served by PicX — AI-generated visual whiteboard summaries of research papers._