Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generation

Summary (Overview)

• ATAS is a world-state-centric framework for long-form agentic video generation that maintains persistent scene memory (static geometry in a shared coordinate frame) and event memory (time-indexed dynamic observations) to enable spatial revisits and historical event recapture across nonconsecutive shots.
• The framework introduces a validation-controlled update mechanism that separates film delivery from state admission—generated clips can be kept in the final film even when their observations are rejected from persistent state, preventing error propagation.
• ATAS compiles screenplays into bounded shot requests with explicit scene, spatial anchor, camera, time interval, and optional event identifiers, using a recursive compilation process.
• Across four video generators (MiniMax-H3, Wan 2.7, HappyHorse 1.0, Wan 3.0), ATAS improves every reported cross-shot consistency measure and achieves the highest aggregate benchmark score for each backbone.
• ATAS outperforms agent-based and 3D-guided methods on spatial revisit (15.968 PSNR, 0.385 SSIM, 0.284 LPIPS), leads three of five event-recapture criteria, and receives the highest human ratings on all six evaluation criteria.

Introduction and Theoretical Foundation

Background and Motivation

Long-form agentic video generation translates a screenplay into a sequence of video shots by coordinating shot planning, reference selection, and video synthesis. As a story unfolds across locations, times, and viewpoints, coherence depends on preserving spatial and temporal relationships established in earlier shots. The paper identifies a key limitation:

"A story may depict an event in a room, shift to another location, and later return to the room or replay the earlier event from a different viewpoint. The return shot should preserve the established scene layout, while the replay should reproduce the same event, including its actions and temporal progression, under the new camera view."

Limitations of Existing Approaches

Existing agentic video systems preserve continuity through:

• Recent frames
• Text summaries
• Retrieved reference images

These cues are effective for locally consecutive shots but fail to:

• Organize past observations in a shared spatial and temporal frame
• Support non-local requests to revisit earlier scenes or events

Key Theoretical Insight

Maintaining a persistent world state introduces two challenges:

1. Addressability: Past observations must remain addressable in both space and time.
2. Reliability: The state must evolve without blindly absorbing generated content whose visual, geometric, or temporal errors could propagate to later shots.

The paper argues that a practical system must:

• Resolve each request into a concrete shot specification
• Retrieve relevant spatial and temporal evidence
• Incorporate observations into persistent state only when sufficiently reliable

Methodology

World State Definition

The persistent world state is defined as the pair of scene and event memories:

$$\mathcal{W}_t = (\mathcal{S}_t, \mathcal{E}_t). \tag{1}$$

Workflow

The overall workflow follows a plan-act-observe loop:

$$(u, \mathcal{W}_t) \xrightarrow{\text{Compile}} q_t \xrightarrow{\text{Query}} C_t \xrightarrow{G} V_t \xrightarrow{\text{Reconstruct/Register/Validate}} (V_t, \mathcal{W}_{t+1}). \tag{2}$$

World-State Compilation

The compiler recursively splits the screenplay by scene, spatial/temporal relations, event references, and camera requirements. Each renderable request is defined as:

$$q_{t,k} = (d_{t,k}, s_{t,k}, \xi_{t,k}, c_{t,k}, \tau_{t,k}, e_{t,k}), \tag{3}$$

where:

• $d_{t,k}$: local shot description
• $s_{t,k}$: scene identifier
• $\xi_{t,k}$: spatial anchor in the scene coordinate frame
• $c_{t,k}$: target camera
• $\tau_{t,k}$: requested time interval
• $e_{t,k}$: optional event identifier

The compiler produces a request schedule:

$$\mathcal{Q}_t = \text{Compile}(u, \mathcal{W}_t) = \{q_{t,1}, \dots, q_{t,K_t}\}. \tag{4}$$

Two-Stage Memory

Scene Memory

The registered source observation $o^{\text{src}}$ establishes the scene coordinate frame and initializes $\mathcal{S}_0$. Later accepted observations are registered to that frame and merged as static point-cloud evidence (with dynamic regions excluded). Scene references are recovered by rendering:

$$C_t^{\text{scene}} = \text{Render}(\mathcal{S}_t, c_t). \tag{5}$$

Event Memory

Event memory stores dynamic evidence separately:

$$\mathcal{E}_t = \{\epsilon_i\}_{i=1}^{N_t}, \qquad \epsilon_i = (s_i, e_i, b_i, \tau_i, \mathcal{D}_i), \tag{6}$$

where $s_i$ is the scene identifier, $e_i$ the event identifier, $b_i$ the source observation, $\tau_i$ the source time interval, and $\mathcal{D}_i$ contains registered frame-level RGB-D, mask, and dynamic point-cloud evidence.

Event retrieval is defined as:

$$C_t^{\text{event}} = \begin{cases} \text{Retrieve}(\mathcal{E}_t, s_t, e_t, \tau_t), & e_t \neq \varnothing, \\ \varnothing, & \text{otherwise}. \end{cases} \tag{7}$$

Shot Generation

The shot prompt and context-dependent prompt condition the generator:

$$C_t = (C_t^{\text{scene}}, C_t^{\text{event}}), \qquad V_t = G(q_t, C_t). \tag{8}$$

Observation Registration and State Update

The generated clip undergoes shot validation with two separate decisions:

• Film review ($f_t$): whether the clip is accepted for delivery
• Registration ($r_t$): whether the reconstruction aligns to the scene coordinate frame

Candidate observation construction:

$$\widetilde{\mathcal{O}}_t = \text{Reconstruct}(V_t), \qquad \mathcal{O}_t = \text{Register}(\widetilde{\mathcal{O}}_t, \mathcal{S}_t). \tag{9}$$

The proposed next state:

$$\widehat{\mathcal{W}}_{t+1} = \text{Update}(\mathcal{W}_t, \mathcal{O}_t). \tag{10}$$

The actual transition rule:

$$\mathcal{W}_{t+1} = \begin{cases} \widehat{\mathcal{W}}_{t+1}, & f_t = r_t = a_t = 1, \\ \mathcal{W}_t, & \text{otherwise}. \end{cases} \tag{11}$$

Key design principle: Film and state update are independent outputs—$V_t$ remains in the film when $f_t = 1$ even if $r_t = 0$ or $a_t = 0$, but the clip is unavailable as evidence for later requests.

Two Persistence Queries

1. Spatial revisit: Emits the earlier scene identifier and requested return anchor/camera; renders $\mathcal{S}_t$ and compares with the target image.
2. Historical event recapture: Includes an earlier event identifier and time interval; retrieves matching records from $\mathcal{E}_t$ and generates the event from the requested camera.

Empirical Validation / Results

Experimental Setup

Benchmarks used:

• VBench: aesthetics, subject/background consistency, temporal flicker, motion smoothness, dynamic degree
• ST-Bench: aesthetics, global/single-shot prompt following, cross-shot consistency (C-All, C-Top10)
• ViStory-Self: identity, appearance, temporal consistency within a story
• ViStory-Cross: identity, appearance, scene consistency across shots

Main Benchmark Results

Table 1: Agent-level quantitative evaluation (excerpt showing best results per backbone)

Bold marks best within each backbone block. ATAS leads every reported aggregate on MiniMax-H3, Wan 3.0, and HappyHorse 1.0, and remains ahead on ST-Bench and both ViStory measures for Wan 2.7.

Spatial Revisit Results

Table 2: Spatial revisit (target-image comparison under common-visibility mask)

Event Recapture Results

Table 3: Source-grounded event recapture (VLM judge, five criteria)

ATAS leads event identity, temporal order, and viewpoint realization, ties ReCA on character/object consistency, and remains slightly lower on action-state preservation (0.688 vs. 0.700).

Human Evaluation

ATAS receives the highest mean rating on all six criteria:

• Script faithfulness: 4.40
• Character consistency: 4.22
• Visual appeal: 4.10
• Narrative coherence: 3.77
• Background consistency: 3.62

Ablation Studies

Table 4: Agent-component ablation (cumulative components, Wan 3.0)

Table 5: Memory analysis (varying stored evidence, Wan 3.0)

Efficiency Analysis

• Retries fall from 1.00 to 0.54 per shot
• Retry-adjusted delivery time declines from 359 to 339 seconds per shot
• Per-story runtime rises from 51.9 to 64.9 minutes
• Memory overhead: 98 seconds per shot
• Throughput falls from 7.28 to 6.21 shots per hour
• Film-review rejection decreases from 52.6% (no memory) to 36.8% (full state)

Theoretical and Practical Implications

Theoretical Contributions

1. World-state-centric paradigm: ATAS reframes agentic video generation as a persistent-state management problem rather than a prompt-chaining problem. The formal separation of scene memory (static geometry) from event memory (dynamic observations) provides a principled decomposition of what must be remembered.

2. Validation-controlled state evolution: The separation of film acceptance ($f_t$) from state admission ($a_t$) is a novel theoretical contribution. Equation 11 formalizes the decision rule, preventing unreliable generated evidence from contaminating future generations while still allowing clips to be delivered as film.

3. Addressability in space and time: The framework demonstrates that explicit spatial anchors ($\xi_t$) and event-time intervals ($\tau_t$) make past observations queryable in ways that implicit context (recent frames, text summaries) cannot achieve.

Practical Implications

• Interactive filmmaking: Directors can request return shots or event replays from new camera angles without regenerating from scratch.
• Episodic content creation: Long-running series can maintain consistent environments and event histories across episodes.
• Previsualization: Filmmakers can explore alternative camera angles on established scenes and events.
• Error containment: The validation mechanism prevents a single bad generation from cascading through the entire film.

Key Findings

• The improvement pattern across open-source and proprietary backbones suggests gains come from persistent multi-shot state rather than any single renderer.
• Scene and event memory contribute complementary improvements: scene memory primarily improves spatial consistency (LPIPS$_C$), while event memory gives the largest recapture gain.
• Recursive compilation amplifies the benefits of memory, suggesting that request decomposition is crucial for leveraging persistent state.
• The remaining difficulty is exact action-state reproduction rather than retrieval of the requested occurrence or view.

Conclusion

ATAS compiles screenplays into shot requests grounded in explicit scene, camera, and event-time references, then updates persistent state only from validated observations. The framework demonstrates:

• Improved cross-shot consistency across four video generators
• Superior spatial revisit performance compared to agent-based and 3D-guided methods
• Strong event recapture capabilities with a source-grounded evaluation protocol
• Highest human ratings on all six evaluation criteria
• Reduced retry demand at a measurable compute cost

The ablations show complementary contributions from scene memory, event memory, and recursive compilation. By keeping locations and prior events addressable throughout a story, ATAS provides a foundation for agentic video tools in interactive filmmaking, episodic content creation, and previsualization.

Future directions implied by the work include:

• Improving exact action-state reproduction in event recapture
• Reducing computational overhead of memory maintenance
• Extending the framework to handle more complex multi-scene interactions
• Addressing environment-wide consistency and whole-story coherence, which received lower human ratings