# VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System

> VideoCoCo uses executable Blender code as chain-of-thought to enforce physical consistency, achieving state-of-the-art on PhyGenBench and VBench-2.0.

- **Source:** [arXiv](https://arxiv.org/abs/2607.27380)
- **Published:** 2026-08-01
- **Permalink:** https://picx.dev/p/DedoDn
- **Whiteboard:** https://picx.dev/p/DedoDn/image

## Summary

## Summary (Overview)

- **VideoCoCo** introduces a novel dual-engine framework for physically consistent video generation that uses **executable Blender code as a process-level chain of thought**, separating spatiotemporal reasoning from visual realization.
- The framework consists of an **executable simulation engine** that synthesizes and runs Blender code to produce a deterministic low-fidelity draft video, and a **generative video engine** that transforms this draft into a photorealistic video via draft-conditioned editing.
- A curated dataset, **VideoCoCo-3K**, provides aligned draft–instruction–target triplets to adapt the video editor to simulated drafts, bridging the gap between simulation-style and photorealistic visuals.
- **VideoCoCo achieves state-of-the-art results** on both PhyGenBench (0.558 average, +0.083 over OmniWeaving baseline) and VBench-2.0 (77.88% average, +25.70 points over baseline), with particularly large gains in material and thermal dynamics.
- The approach is fully **automatic, inspectable, and reproducible**, as every intermediate artifact—code, draft video, and editing instruction—is explicitly represented and can be examined or revised.

## Introduction and Theoretical Foundation

The paper addresses a fundamental limitation in text-to-video generation: the **Causal Opacity** problem, where a text prompt compresses a physical event into a few words, forcing the model to implicitly reconstruct the entire spatiotemporal process while simultaneously synthesizing visual appearance. This mismatch between compressed intent and fully realized evolution leads to physically inconsistent dynamics.

The authors argue that existing chain-of-thought (CoT) approaches for video generation fall short because their intermediate representations are either:
- **Non-executable** (textual plans, layouts, keyframes)
- **Temporally sparse** (isolated keyframes)
- **Not directly inspectable or revisable** (hidden in latent activations)

The key insight is that **executable code itself can serve as a process-level intermediate representation**. Unlike prior methods, code is:
1. **Explicit**: every object, motion, and interaction must be declared
2. **Executable**: the program commits to a concrete process that can actually be run
3. **Inspectable**: the code can be read, edited, and re-executed

The theoretical foundation builds on extending Code-as-CoT reasoning from image preview (as in CoCo [42]) to complete temporal processes for video generation.

## Methodology

### Dual-Engine Architecture

The VideoCoCo framework consists of two complementary engines:

**1. Executable Simulation Engine**
- Given a text prompt $p$, a coding agent $A_{\text{code}}$ synthesizes a self-contained Blender Python program:
  $$c = A_{\text{code}}(p)$$
- The program is executed in a sandboxed Blender environment $B$ to produce a deterministic spatiotemporal draft:
  $$d = B(c)$$
- The draft is intentionally **low-fidelity** (white-clay simulation without photorealistic materials or complex lighting) but **temporally dense**, with every frame corresponding to a physically instantiated state.

**2. Generative Video Engine**
- An instruction agent $A_{\text{edit}}$ reads both the prompt $p$ and draft $d$ to compose an appearance-focused editing instruction:
  $$e = A_{\text{edit}}(p, d)$$
- The instruction describes subjects, materials, lighting, and cinematic style, but is **discouraged from redefining motion** already present in the draft.
- A draft-conditioned video editor $G_\theta$ produces the final video:
  $$\hat{v} = G_\theta(d, e)$$

### VideoCoCo-3K Dataset

To adapt the video editor to simulated drafts, the authors construct a dataset of **draft–instruction–target triplets**:

$$D_{\text{VideoCoCo-3K}} = \{(d_i, e_i, y_i)\}_{i=1}^{3000}$$

where the photorealistic target $y_i$ is generated by a high-fidelity teacher editor $G_T$ (Seedance 2.0):
$$y_i = G_T(d_i, e_i)$$

### Training Objective

The draft-conditioned editor $G_\theta$ is adapted using a standard conditional denoising objective:
$$\mathcal{L}(\theta) = \mathbb{E}_{(d,e,y), t, \epsilon}\left[\|\epsilon - \epsilon_\theta(z_t, t, d, e)\|_2^2\right]$$

where $z_0$ is the latent of target $y$, $z_t$ is its noised version at timestep $t$, and $\epsilon \sim \mathcal{N}(0, I)$.

Three adaptation strategies are compared:
- **Tuning-Free**: direct conditioning without parameter updates
- **Full Fine-Tuning**: updating all parameters
- **LoRA Tuning**: parameter-efficient low-rank adaptation

## Empirical Validation / Results

### PhyGenBench Results (Table 1)

| Method | Mechanics (↑) | Optics (↑) | Thermal (↑) | Material (↑) | Average (↑) |
|--------|:------------:|:---------:|:-----------:|:-----------:|:-----------:|
| OmniWeaving | 0.48 | 0.56 | 0.43 | 0.39 | 0.48 |
| **+ VideoCoCo** | **0.56** | **0.61** | **0.51** | **0.53** | **0.56** |
| Wan2.2-TI2V-5B | 0.55 | 0.58 | 0.53 | 0.50 | 0.54 |

Key findings:
- VideoCoCo achieves the **best average score (0.558)** among all systems, surpassing Wan2.2-TI2V-5B (0.544)
- Largest gains over OmniWeaving base: **material (+0.133)** and **thermal (+0.078)**
- These are precisely the categories where appearance-driven generators are weakest, confirming that code contributes physical dynamics rather than surface realism

### VBench-2.0 Results (Table 2)

| Method | Mechanics (↑) | Thermotics (↑) | Material (↑) | Average (↑) |
|--------|:------------:|:-------------:|:-----------:|:-----------:|
| OmniWeaving | 62.79% | 52.08% | 41.67% | 52.18% |
| **+ VideoCoCo** | **92.31%** | **72.92%** | 68.42% | **77.88%** |
| CogVideoX-1.5 | 80.80% | 67.13% | **83.19%** | 77.04% |

Key findings:
- **+25.70 points gain** over OmniWeaving baseline (52.18% → 77.88%)
- Best mechanics (92.31%) and thermotics (72.92%) scores among all systems
- Second on material (68.42%), trailing only CogVideoX-1.5 (83.19%)

### Ablation Study (Table 3)

| Method | Mech. (↑) | Opt. (↑) | Therm. (↑) | Mat. (↑) | Avg. (↑) |
|--------|:---------:|:--------:|:----------:|:--------:|:--------:|
| OmniWeaving | 0.48 | 0.56 | 0.43 | 0.39 | 0.48 |
| + VideoCoCo (Tune-Free) | 0.50 | 0.53 | 0.48 | 0.51 | **0.51** |
| + VideoCoCo (Full-Tune) | 0.51 | 0.61 | 0.51 | 0.49 | **0.54** |
| + VideoCoCo (LoRA-Tune) | 0.56 | 0.61 | 0.51 | 0.53 | **0.56** |

Key findings:
- **Tuning-Free variant** already surpasses OmniWeaving (0.48 → 0.51), isolating the contribution of executable drafting
- **LoRA tuning** outperforms full fine-tuning (0.558 vs. 0.535) despite fewer parameters, suggesting the adaptation task is narrow and benefits from retaining strong visual priors
- Both drafting and editor tuning contribute complementary gains

## Theoretical and Practical Implications

**Theoretical Implications:**
- The work demonstrates that **executable code is an effective process-level intermediate representation** for video generation, providing a principled way to externalize physical reasoning before pixel synthesis
- The **dual-engine decomposition** separates two fundamentally different challenges: deterministic physical simulation (well-suited to code agents and physics engines) and photorealistic appearance (well-suited to generative models)
- The concept of **Code-as-CoT** extends from image preview to complete temporal processes, suggesting a general paradigm for incorporating structured reasoning into generative tasks

**Practical Implications:**
- VideoCoCo is **fully automatic, inspectable, and reproducible**—every intermediate artifact (code, draft, instruction) can be examined and revised
- The approach is **model-agnostic** and can augment any base video generator (demonstrated with OmniWeaving)
- The **VideoCoCo-3K dataset** provides aligned supervision that can be extended or regenerated as the pipeline evolves
- **Lightweight LoRA adaptation** is sufficient and preferable to full fine-tuning, making the approach computationally practical

**Limitations:**
- Additional inference latency due to code synthesis and sandbox execution
- Bounded by the expressiveness of the Blender simulator—complex phenomena like turbulent fluids remain challenging zero-shot
- Future work may integrate specialized physics engines (e.g., Taichi) and explore knowledge distillation to internalize executable priors into end-to-end models

## Conclusion

VideoCoCo introduces an agentic dual-engine framework that uses **executable Blender code as a process-level chain of thought** to enforce physical consistency in text-to-video generation. By decoupling spatiotemporal simulation from photorealistic editing, the framework achieves **state-of-the-art physical plausibility** on both PhyGenBench (0.558 average) and VBench-2.0 (77.88% average), with particularly large gains in material and thermal dynamics where appearance-driven priors are weakest.

The key contributions are:
1. A novel **Code-as-CoT paradigm** for video generation that externalizes physical reasoning as executable simulation code
2. The **VideoCoCo-3K dataset** providing aligned draft–instruction–target triplets for draft-conditioned editing
3. Empirical validation that **both drafting and editor tuning contribute complementary gains**, with lightweight LoRA adaptation being sufficient

The work demonstrates that executable code provides an **effective, controllable, and inspectable intermediate representation** for physically consistent video generation, opening new directions for integrating structured reasoning with generative models.

---

_Markdown view of https://picx.dev/p/DedoDn, served by PicX — AI-generated visual whiteboard summaries of research papers._