Summary (Overview)

• VideoCoCo introduces a novel dual-engine framework for physically consistent video generation that uses executable Blender code as a process-level chain of thought, separating spatiotemporal reasoning from visual realization.
• The framework consists of an executable simulation engine that synthesizes and runs Blender code to produce a deterministic low-fidelity draft video, and a generative video engine that transforms this draft into a photorealistic video via draft-conditioned editing.
• A curated dataset, VideoCoCo-3K, provides aligned draft–instruction–target triplets to adapt the video editor to simulated drafts, bridging the gap between simulation-style and photorealistic visuals.
• VideoCoCo achieves state-of-the-art results on both PhyGenBench (0.558 average, +0.083 over OmniWeaving baseline) and VBench-2.0 (77.88% average, +25.70 points over baseline), with particularly large gains in material and thermal dynamics.
• The approach is fully automatic, inspectable, and reproducible, as every intermediate artifact—code, draft video, and editing instruction—is explicitly represented and can be examined or revised.

Introduction and Theoretical Foundation

The paper addresses a fundamental limitation in text-to-video generation: the Causal Opacity problem, where a text prompt compresses a physical event into a few words, forcing the model to implicitly reconstruct the entire spatiotemporal process while simultaneously synthesizing visual appearance. This mismatch between compressed intent and fully realized evolution leads to physically inconsistent dynamics.

The authors argue that existing chain-of-thought (CoT) approaches for video generation fall short because their intermediate representations are either:

• Non-executable (textual plans, layouts, keyframes)
• Temporally sparse (isolated keyframes)
• Not directly inspectable or revisable (hidden in latent activations)

The key insight is that executable code itself can serve as a process-level intermediate representation. Unlike prior methods, code is:

1. Explicit: every object, motion, and interaction must be declared
2. Executable: the program commits to a concrete process that can actually be run
3. Inspectable: the code can be read, edited, and re-executed

The theoretical foundation builds on extending Code-as-CoT reasoning from image preview (as in CoCo [42]) to complete temporal processes for video generation.

Methodology

Dual-Engine Architecture

The VideoCoCo framework consists of two complementary engines:

1. Executable Simulation Engine

• Given a text prompt $p$, a coding agent $A_{\text{code}}$ synthesizes a self-contained Blender Python program: $$c = A_{\text{code}}(p)$$
• The program is executed in a sandboxed Blender environment $B$ to produce a deterministic spatiotemporal draft: $$d = B(c)$$
• The draft is intentionally low-fidelity (white-clay simulation without photorealistic materials or complex lighting) but temporally dense, with every frame corresponding to a physically instantiated state.

2. Generative Video Engine

• An instruction agent $A_{\text{edit}}$ reads both the prompt $p$ and draft $d$ to compose an appearance-focused editing instruction: $$e = A_{\text{edit}}(p, d)$$
• The instruction describes subjects, materials, lighting, and cinematic style, but is discouraged from redefining motion already present in the draft.
• A draft-conditioned video editor $G_\theta$ produces the final video: $$\hat{v} = G_\theta(d, e)$$

VideoCoCo-3K Dataset

To adapt the video editor to simulated drafts, the authors construct a dataset of draft–instruction–target triplets:

$$D_{\text{VideoCoCo-3K}} = \{(d_i, e_i, y_i)\}_{i=1}^{3000}$$

where the photorealistic target $y_i$ is generated by a high-fidelity teacher editor $G_T$ (Seedance 2.0):

$$y_i = G_T(d_i, e_i)$$

Training Objective

The draft-conditioned editor $G_\theta$ is adapted using a standard conditional denoising objective:

$$\mathcal{L}(\theta) = \mathbb{E}_{(d,e,y), t, \epsilon}\left[\|\epsilon - \epsilon_\theta(z_t, t, d, e)\|_2^2\right]$$

where $z_0$ is the latent of target $y$, $z_t$ is its noised version at timestep $t$, and $\epsilon \sim \mathcal{N}(0, I)$.

Three adaptation strategies are compared:

• Tuning-Free: direct conditioning without parameter updates
• Full Fine-Tuning: updating all parameters
• LoRA Tuning: parameter-efficient low-rank adaptation

Empirical Validation / Results

PhyGenBench Results (Table 1)

Key findings:

• VideoCoCo achieves the best average score (0.558) among all systems, surpassing Wan2.2-TI2V-5B (0.544)
• Largest gains over OmniWeaving base: material (+0.133) and thermal (+0.078)
• These are precisely the categories where appearance-driven generators are weakest, confirming that code contributes physical dynamics rather than surface realism

VBench-2.0 Results (Table 2)

Key findings:

• +25.70 points gain over OmniWeaving baseline (52.18% → 77.88%)
• Best mechanics (92.31%) and thermotics (72.92%) scores among all systems
• Second on material (68.42%), trailing only CogVideoX-1.5 (83.19%)

Ablation Study (Table 3)

Key findings:

• Tuning-Free variant already surpasses OmniWeaving (0.48 → 0.51), isolating the contribution of executable drafting
• LoRA tuning outperforms full fine-tuning (0.558 vs. 0.535) despite fewer parameters, suggesting the adaptation task is narrow and benefits from retaining strong visual priors
• Both drafting and editor tuning contribute complementary gains

Theoretical and Practical Implications

Theoretical Implications:

• The work demonstrates that executable code is an effective process-level intermediate representation for video generation, providing a principled way to externalize physical reasoning before pixel synthesis
• The dual-engine decomposition separates two fundamentally different challenges: deterministic physical simulation (well-suited to code agents and physics engines) and photorealistic appearance (well-suited to generative models)
• The concept of Code-as-CoT extends from image preview to complete temporal processes, suggesting a general paradigm for incorporating structured reasoning into generative tasks

Practical Implications:

• VideoCoCo is fully automatic, inspectable, and reproducible—every intermediate artifact (code, draft, instruction) can be examined and revised
• The approach is model-agnostic and can augment any base video generator (demonstrated with OmniWeaving)
• The VideoCoCo-3K dataset provides aligned supervision that can be extended or regenerated as the pipeline evolves
• Lightweight LoRA adaptation is sufficient and preferable to full fine-tuning, making the approach computationally practical

Limitations:

• Additional inference latency due to code synthesis and sandbox execution
• Bounded by the expressiveness of the Blender simulator—complex phenomena like turbulent fluids remain challenging zero-shot
• Future work may integrate specialized physics engines (e.g., Taichi) and explore knowledge distillation to internalize executable priors into end-to-end models

Conclusion

VideoCoCo introduces an agentic dual-engine framework that uses executable Blender code as a process-level chain of thought to enforce physical consistency in text-to-video generation. By decoupling spatiotemporal simulation from photorealistic editing, the framework achieves state-of-the-art physical plausibility on both PhyGenBench (0.558 average) and VBench-2.0 (77.88% average), with particularly large gains in material and thermal dynamics where appearance-driven priors are weakest.

The key contributions are:

1. A novel Code-as-CoT paradigm for video generation that externalizes physical reasoning as executable simulation code
2. The VideoCoCo-3K dataset providing aligned draft–instruction–target triplets for draft-conditioned editing
3. Empirical validation that both drafting and editor tuning contribute complementary gains, with lightweight LoRA adaptation being sufficient

The work demonstrates that executable code provides an effective, controllable, and inspectable intermediate representation for physically consistent video generation, opening new directions for integrating structured reasoning with generative models.