# FrameForge: Harnessing Priors of Video Foundation Models for Unified Image Generation

> FrameForge unifies subject-driven, canny-to-image, and editing tasks in one model by adapting video foundation priors via input-adaptive virtual temporal relations.

- **Source:** [arXiv](https://arxiv.org/abs/2610.36545)
- **Published:** 2026-11-03
- **Permalink:** https://picx.dev/p/XFUixy
- **Whiteboard:** https://picx.dev/p/XFUixy/image

## Summary

# FrameForge: Harnessing Priors of Video Foundation Models for Unified Image Generation

**Authors:** Yijing Lin, Zhendong Mao, Mengqi Huang, Yuxin Wang, Yongdong Zhang  
**Affiliations:** University of Science and Technology of China, FrameX.AI

---

## Summary (Overview)

- **Novel framework for unified conditional image generation**: FrameForge adapts a pretrained video foundation model (HunyuanVideo1.5) to handle heterogeneous conditional image generation tasks (subject-driven generation, canny-to-image generation, and instruction-guided image editing) within a single unified model using only one shared LoRA adapter.

- **Key insight**: Different conditional generation tasks exhibit distinct condition-target transformation magnitudes (measured by LPIPS distance) that substantially overlap with those observed in natural video transitions, motivating the modeling of conditional generation through *input-adaptive virtual temporal relations* rather than fixed temporal offsets.

- **Two complementary technical contributions**: Mixture of Temporal Experts (MoTE) dynamically selects among temporal RoPE experts with different virtual intervals, while Chain of Temporal Reasoning (CoTR) inserts a noise-initialized latent with a reasoning-bottleneck attention mask to model implicit condition-to-target transformations.

- **State-of-the-art results**: FrameForge achieves superior performance on DreamBench (subject-driven: O=0.739), COCO2017 (canny-to-image: SSIM=0.447), and GEdit-Bench (editing: G-O=6.12 overall, 6.84 on Motion Change), outperforming both task-specific and unified baselines.

- **Extensive validation**: Comprehensive ablations verify the complementarity of MoTE and CoTR, and the framework generalizes across different video backbones (HunyuanVideo1.0 and 1.5).

---

## Introduction and Theoretical Foundation

### Background and Motivation

Real-world image generation involves diverse inputs (text, reference images, structural maps) that impose constraints at different semantic and spatial levels. Modeling these heterogeneous condition-target relations within a unified framework is challenging because:

- **Image-centric approaches** (e.g., OminiControl, ACE++, EasyControl) inject conditions as auxiliary spatial or semantic signals without explicitly modeling how strongly the condition and target differ or how one transforms into the other.
- **Video-model-based approaches** (e.g., RealGeneral, DRA-Ctrl, ChronoEdit) assign condition and target to *fixed* temporal positions, resulting in an invariant temporal positional relation across tasks and samples.

### Key Observation

> In natural videos, frames separated by different temporal intervals exhibit different degrees of visual variation. A similar phenomenon arises in conditional image generation: image editing typically requires relatively small changes while preserving most source content, whereas subject-driven and canny-to-image generation often involve larger variations.

The paper demonstrates (Fig. 1) that different conditional generation tasks exhibit clearly different condition-target LPIPS distributions, and these distributions substantially overlap with those observed in natural video transitions. This suggests that different tasks correspond to different regions of the temporal transformation prior learned by a video model.

### Theoretical Formulation

The authors propose modeling conditional generation through an **input-adaptive virtual temporal interval** rather than a fixed offset. This perspective reveals two complementary aspects:

1. **How far** the target lies from the condition under the learned temporal prior (addressed by MoTE)
2. **How** the condition should be transformed into the target (addressed by CoTR)

---

## Methodology

### Framework Architecture

FrameForge is built upon HunyuanVideo1.5's first-stage video diffusion model (MM-DiT backbone with 3D VAE):

- **VAE encoding**: Compresses spatial dimensions by 16× and temporal by 4×. Given video $\mathbf{x} \in \mathbb{R}^{(4f+1) \times 16h \times 16w \times c}$, the encoded latent is:
  $$\mathbf{z}_0 = \mathcal{E}(\mathbf{x}) \in \mathbb{R}^{(f+1) \times h \times w \times 32}$$
  
- **Flow matching objective**:
  $$\mathcal{L}_{\mathrm{diffusion}} = \left\| \mathbf{v} - \mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{c}) \right\|_2^2$$

- **Overall conditioning**:
  $$\mathbf{c}^{\mathrm{ours}} = \left[ \mathbf{z}^{\mathrm{text}}, \mathbf{z}^{\mathrm{cond}}, \mathbf{z}^{\mathrm{reason}} \right]$$

### Mixture of Temporal Experts (MoTE)

MoTE constructs $E$ temporal positional experts with candidate latent spans $\mathcal{S} = \{s_1, s_2, \ldots, s_E\}$, where each expert corresponds to a virtual temporal interval $\Delta_e = s_e - 1$. The default configuration uses video-frame spans {17, 69, 121} corresponding to latent spans {5, 18, 31}.

**Routing mechanism**:
- A lightweight router $G$ takes concatenated text, condition, and noisy target representations to produce token-level routing logits:
  $$\mathbf{A} = G\left(\left[\mathbf{z}^{\mathrm{text}}, \mathbf{z}^{\mathrm{cond}}, \mathbf{z}_t^{\mathrm{target}}\right]\right) \in \mathbb{R}^{L \times E}$$
- Masked average pooling produces sample-level logits, with routing probabilities:
  $$\mathbf{p} = \operatorname{Softmax}\left(\frac{\mathbf{a}}{\tau_{\text{router}}}\right), \qquad e^* = \underset{e \in \{1,\dots,E\}}{\arg\max}\, p_e$$
- **Straight-through estimator** for end-to-end optimization with discrete routing:
  $$\mathbf{g} = \operatorname{sg}\left(\mathbf{g}^{\mathrm{hard}} - \mathbf{p}\right) + \mathbf{p}$$

**Load-balancing regularization**:
$$\mathcal{L}_{\mathrm{balance}} = -\sum_{e=1}^{E} \log(\bar{g}_e + \epsilon)$$

**Final objective**:
$$\mathcal{L}_{\mathrm{final}} = \mathcal{L}_{\mathrm{diffusion}} + \lambda_{\mathrm{bal}} \mathcal{L}_{\mathrm{balance}}$$

### Chain of Temporal Reasoning (CoTR)

CoTR inserts a single noise-initialized reasoning latent $\mathbf{z}^{\mathrm{reason}}$ between condition and target to model implicit transformations:

- **Reasoning-bottleneck attention mask** (applied with probability $p_{\mathrm{mask}} = 0.5$ during intermediate training):
  $$\mathbf{M}_{ij}^{\mathrm{cotr}} = \begin{cases} -\infty, & i \in \mathcal{T} \cup \mathcal{C},\ j \in \mathcal{X} \\ -\infty, & i \in \mathcal{X},\ j \in \mathcal{T} \cup \mathcal{C} \\ 0, & \text{otherwise} \end{cases}$$
  where $\mathcal{T}, \mathcal{C}, \mathcal{R}, \mathcal{X}$ denote text, condition, reasoning, and target token sets.

This mask forces target tokens to receive conditional information through the evolving reasoning representation rather than direct attention paths.

### Three-Stage Training Strategy

1. **Stage 1**: Optimize only the MoTE router (condition-target formulation, no reasoning latent)
2. **Stage 2**: Add reasoning latent, jointly optimize router and LoRA with stochastic attention mask
3. **Stage 3**: Full attention configuration matching inference-time behavior

The video backbone and VAE encoder remain frozen throughout; only a single shared LoRA (rank=32, alpha=32, 182M parameters) is trained.

---

## Empirical Validation / Results

### Training Setup

- **Data**: 170K samples per task (510K total) from Subject200K/UNO-1M (subject-driven), 512-2M (canny-to-image), and GPT-Image-Edit-1.5M (editing)
- **Hardware**: 2× NVIDIA A800 GPUs, 12,000 iterations, batch size 60, learning rate $3 \times 10^{-4}$, ~6 days
- **Inference**: 50 denoising steps; CFG scale 6 (subject-driven, editing), 2 (canny-to-image)

### Subject-Driven Generation (DreamBench)

| Method | Base Model | SC↑ | PQ↑ | O↑ |
|--------|-----------|-----|-----|-----|
| IP-Adapter | SDXL | 0.516 | 0.824 | 0.629 |
| OminiControl | FLUX.1-dev | 0.680 | 0.816 | 0.719 |
| EchoGen | Infinity-2B | 0.729 | 0.744 | 0.719 |
| ChronoEdit | Wan2.1-I2V-14B | 0.757 | 0.764 | 0.729 |
| **Ours** | **HunyuanVideo1.5** | **0.739** | **0.783** | **0.739** |

FrameForge achieves the best overall score (O=0.739), with subject consistency ranked second only to ChronoEdit while achieving notably higher perceptual quality.

### Canny-to-Image Generation (COCO2017)

| Method | Base Model | FID↓ | SSIM↑ | CLIP-T↑ |
|--------|-----------|------|-------|---------|
| ControlNet | SD1.5 | 20.42 | 0.299 | 0.323 |
| EasyControl | FLUX.1-dev | 22.82 | 0.417 | 0.319 |
| RealGeneral | CogVideoX-1.5 | 24.63 | 0.414 | 0.324 |
| DRA-Ctrl | HunyuanVideo-I2V | 23.27 | 0.408 | 0.314 |
| **Ours** | **HunyuanVideo1.5** | **23.47** | **0.447** | **0.323** |

FrameForge achieves the highest SSIM (0.447, 7.2% relative improvement over second-best), indicating superior structural fidelity while maintaining text-image alignment.

### Instruction-Guided Image Editing (GEdit-Bench)

| Method | Backbone | Motion G-O↑ | Overall G-O↑ |
|--------|----------|-------------|--------------|
| GPT-4o (closed) | Native | 7.81 | 8.01 |
| Gemini 2.5 (closed) | Native | 6.72 | 7.89 |
| ChronoEdit | Wan2.1-I2V-14B | 5.76 | 5.08 |
| **Ours** | **HunyuanVideo1.5** | **6.84** | **6.12** |

FrameForge achieves the best open-source performance, surpassing closed-source Gemini 2.5 on Motion Change and approaching GPT-4o.

### MoTE Routing Analysis

The router exhibits **task-dependent and timestep-dependent preferences**:
- Canny-to-image (largest condition-target distance): predominantly selects long-interval expert, especially during early denoising
- Subject-driven and editing: initially favor medium-interval expert, shifting to short-interval as target content emerges
- Editing transitions earlier and more strongly to short intervals, consistent with greater source-content preservation

### Component Ablation Results

| Variant | LoRA | MoTE | CoTR | Subject↑ | Canny↓ | Editing↑ |
|---------|------|------|------|----------|--------|----------|
| Task-specific | Per-task | ✕ | ✕ | 0.676 | 25.22 | 5.99 |
| Unified baseline | Shared | ✕ | ✕ | 0.722 | 25.78 | 4.94 |
| + MoTE | Shared | ✓ | ✕ | 0.719 | 25.62 | 5.77 |
| + CoTR | Shared | ✕ | ✓ | 0.726 | 25.94 | 5.56 |
| **Full Model** | **Shared** | **✓** | **✓** | **0.739** | **23.47** | **6.12** |

### Cross-Backbone Generalization

On HunyuanVideo1.0, FrameForge improves subject-driven (0.699→0.736), editing (4.38→5.00), and canny FID (24.52→23.07) over unified baseline, confirming generalizability.

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **New perspective on conditional generation**: The paper reframes heterogeneous conditional image generation as a problem of *input-adaptive virtual temporal relations* in video models, providing a principled way to unify tasks with vastly different transformation magnitudes.

2. **Complementary modeling of "how far" and "how"**: MoTE addresses the distance aspect through adaptive temporal positional priors, while CoTR addresses the transformation pathway through an unsupervised reasoning latent. The ablation study confirms they are complementary—neither alone recovers task-specific baseline performance, but their combination surpasses it.

3. **Video priors as general foundations**: The results provide strong evidence that video foundation models encode transferable priors for structured visual transformations applicable to static image generation.

### Practical Implications

- **Unified model, single adapter**: A single shared LoRA (182M parameters) handles three heterogeneous tasks, significantly reducing deployment overhead compared to task-specific models
- **No task-specific adaptation needed**: The MoTE router automatically discovers appropriate temporal relations per task and per denoising step
- **Broad applicability**: The framework generalizes across different video backbones (HunyuanVideo1.0 and 1.5), suggesting the approach is backbone-agnostic
- **Outperforms closed-source systems on motion editing**: FrameForge surpasses Gemini 2.5 and Seedream 4.0 on Motion Change editing, demonstrating the value of video-based temporal priors for dynamic edits

---

## Conclusion

FrameForge introduces a unified adaptation framework that leverages pretrained video foundation models for heterogeneous conditional image generation. The key contributions are:

1. **Conceptual**: Formulating conditional generation through input-adaptive virtual temporal relations, motivated by the observation that task LPIPS distances overlap with natural video transition distributions

2. **Technical**: 
   - **MoTE**: Dynamically selects temporal RoPE experts based on text and visual context, adapting the virtual interval between condition and target per sample and denoising step
   - **CoTR**: Inserts an unsupervised reasoning latent with a bottleneck attention mask to model condition-to-target transformations without explicit intermediate supervision

3. **Experimental**: Superior performance across DreamBench, COCO2017, and GEdit-Bench with a single unified model, comprehensive ablations validating both components, and demonstrated generalization across backbones

### Future Directions

While the paper doesn't explicitly enumerate future work, the findings suggest several promising avenues: exploring additional tasks within the unified framework (e.g., depth-to-image, inpainting), investigating the interpretability of the reasoning latent, and extending the approach to video-to-video generation where temporal relations are more complex.

---

_Markdown view of https://picx.dev/p/XFUixy, served by PicX — AI-generated visual whiteboard summaries of research papers._