# Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generation

> ATAS maintains persistent scene and event memory with validation-controlled updates, enabling spatial revisits and event recaptures that improve cross-shot consistency across four video generators.

- **Source:** [arXiv](https://arxiv.org/abs/2610.20571)
- **Published:** 2026-10-16
- **Permalink:** https://picx.dev/p/3RYm0O
- **Whiteboard:** https://picx.dev/p/3RYm0O/image

## Summary

# Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generation

## Summary (Overview)

- **ATAS** is a world-state-centric framework for long-form agentic video generation that maintains persistent **scene memory** (static geometry in a shared coordinate frame) and **event memory** (time-indexed dynamic observations) to enable spatial revisits and historical event recapture across nonconsecutive shots.
- The framework introduces a **validation-controlled update mechanism** that separates film delivery from state admission—generated clips can be kept in the final film even when their observations are rejected from persistent state, preventing error propagation.
- ATAS compiles screenplays into **bounded shot requests** with explicit scene, spatial anchor, camera, time interval, and optional event identifiers, using a recursive compilation process.
- Across **four video generators** (MiniMax-H3, Wan 2.7, HappyHorse 1.0, Wan 3.0), ATAS improves every reported cross-shot consistency measure and achieves the highest aggregate benchmark score for each backbone.
- ATAS outperforms agent-based and 3D-guided methods on **spatial revisit** (15.968 PSNR, 0.385 SSIM, 0.284 LPIPS), leads three of five event-recapture criteria, and receives the highest human ratings on all six evaluation criteria.

## Introduction and Theoretical Foundation

### Background and Motivation

Long-form agentic video generation translates a screenplay into a sequence of video shots by coordinating shot planning, reference selection, and video synthesis. As a story unfolds across locations, times, and viewpoints, coherence depends on preserving spatial and temporal relationships established in earlier shots. The paper identifies a key limitation:

> "A story may depict an event in a room, shift to another location, and later return to the room or replay the earlier event from a different viewpoint. The return shot should preserve the established scene layout, while the replay should reproduce the same event, including its actions and temporal progression, under the new camera view."

### Limitations of Existing Approaches

Existing agentic video systems preserve continuity through:
- Recent frames
- Text summaries
- Retrieved reference images

These cues are effective for **locally consecutive shots** but fail to:
- Organize past observations in a shared spatial and temporal frame
- Support **non-local requests** to revisit earlier scenes or events

### Key Theoretical Insight

Maintaining a persistent world state introduces two challenges:
1. **Addressability**: Past observations must remain addressable in both space and time.
2. **Reliability**: The state must evolve without blindly absorbing generated content whose visual, geometric, or temporal errors could propagate to later shots.

The paper argues that a practical system must:
- Resolve each request into a concrete shot specification
- Retrieve relevant spatial and temporal evidence
- Incorporate observations into persistent state only when sufficiently reliable

## Methodology

### World State Definition

The persistent world state is defined as the pair of scene and event memories:

$$
\mathcal{W}_t = (\mathcal{S}_t, \mathcal{E}_t). \tag{1}
$$

### Workflow

The overall workflow follows a plan-act-observe loop:

$$
(u, \mathcal{W}_t) \xrightarrow{\text{Compile}} q_t \xrightarrow{\text{Query}} C_t \xrightarrow{G} V_t \xrightarrow{\text{Reconstruct/Register/Validate}} (V_t, \mathcal{W}_{t+1}). \tag{2}
$$

### World-State Compilation

The compiler recursively splits the screenplay by scene, spatial/temporal relations, event references, and camera requirements. Each renderable request is defined as:

$$
q_{t,k} = (d_{t,k}, s_{t,k}, \xi_{t,k}, c_{t,k}, \tau_{t,k}, e_{t,k}), \tag{3}
$$

where:
- $d_{t,k}$: local shot description
- $s_{t,k}$: scene identifier
- $\xi_{t,k}$: spatial anchor in the scene coordinate frame
- $c_{t,k}$: target camera
- $\tau_{t,k}$: requested time interval
- $e_{t,k}$: optional event identifier

The compiler produces a request schedule:

$$
\mathcal{Q}_t = \text{Compile}(u, \mathcal{W}_t) = \{q_{t,1}, \dots, q_{t,K_t}\}. \tag{4}
$$

### Two-Stage Memory

#### Scene Memory

The registered source observation $o^{\text{src}}$ establishes the scene coordinate frame and initializes $\mathcal{S}_0$. Later accepted observations are registered to that frame and merged as static point-cloud evidence (with dynamic regions excluded). Scene references are recovered by rendering:

$$
C_t^{\text{scene}} = \text{Render}(\mathcal{S}_t, c_t). \tag{5}
$$

#### Event Memory

Event memory stores dynamic evidence separately:

$$
\mathcal{E}_t = \{\epsilon_i\}_{i=1}^{N_t}, \qquad \epsilon_i = (s_i, e_i, b_i, \tau_i, \mathcal{D}_i), \tag{6}
$$

where $s_i$ is the scene identifier, $e_i$ the event identifier, $b_i$ the source observation, $\tau_i$ the source time interval, and $\mathcal{D}_i$ contains registered frame-level RGB-D, mask, and dynamic point-cloud evidence.

Event retrieval is defined as:

$$
C_t^{\text{event}} = \begin{cases} \text{Retrieve}(\mathcal{E}_t, s_t, e_t, \tau_t), & e_t \neq \varnothing, \\ \varnothing, & \text{otherwise}. \end{cases} \tag{7}
$$

### Shot Generation

The shot prompt and context-dependent prompt condition the generator:

$$
C_t = (C_t^{\text{scene}}, C_t^{\text{event}}), \qquad V_t = G(q_t, C_t). \tag{8}
$$

### Observation Registration and State Update

The generated clip undergoes **shot validation** with two separate decisions:
- **Film review** ($f_t$): whether the clip is accepted for delivery
- **Registration** ($r_t$): whether the reconstruction aligns to the scene coordinate frame

Candidate observation construction:

$$
\widetilde{\mathcal{O}}_t = \text{Reconstruct}(V_t), \qquad \mathcal{O}_t = \text{Register}(\widetilde{\mathcal{O}}_t, \mathcal{S}_t). \tag{9}
$$

The proposed next state:

$$
\widehat{\mathcal{W}}_{t+1} = \text{Update}(\mathcal{W}_t, \mathcal{O}_t). \tag{10}
$$

The actual transition rule:

$$
\mathcal{W}_{t+1} = \begin{cases} \widehat{\mathcal{W}}_{t+1}, & f_t = r_t = a_t = 1, \\ \mathcal{W}_t, & \text{otherwise}. \end{cases} \tag{11}
$$

**Key design principle**: Film and state update are independent outputs—$V_t$ remains in the film when $f_t = 1$ even if $r_t = 0$ or $a_t = 0$, but the clip is unavailable as evidence for later requests.

### Two Persistence Queries

1. **Spatial revisit**: Emits the earlier scene identifier and requested return anchor/camera; renders $\mathcal{S}_t$ and compares with the target image.
2. **Historical event recapture**: Includes an earlier event identifier and time interval; retrieves matching records from $\mathcal{E}_t$ and generates the event from the requested camera.

## Empirical Validation / Results

### Experimental Setup

**Benchmarks used:**
- **VBench**: aesthetics, subject/background consistency, temporal flicker, motion smoothness, dynamic degree
- **ST-Bench**: aesthetics, global/single-shot prompt following, cross-shot consistency (C-All, C-Top10)
- **ViStory-Self**: identity, appearance, temporal consistency within a story
- **ViStory-Cross**: identity, appearance, scene consistency across shots

### Main Benchmark Results

**Table 1: Agent-level quantitative evaluation** (excerpt showing best results per backbone)

| Model | Method | VBench | ST-Bench | ViStory-Self | ViStory-Cross | Avg. |
|-------|--------|--------|----------|--------------|---------------|------|
| MiniMax-H3 | ATAS | 0.860 ± 0.010 | **0.691 ± 0.014** | **0.710 ± 0.020** | **0.547 ± 0.021** | **0.702** |
| Wan 2.7 | ATAS | 0.846 ± 0.012 | **0.625 ± 0.016** | **0.666 ± 0.021** | **0.486 ± 0.032** | **0.656** |
| HappyHorse 1.0 | ATAS | 0.862 ± 0.009 | **0.668 ± 0.015** | **0.706 ± 0.019** | **0.540 ± 0.030** | **0.694** |
| Wan 3.0 | ATAS | **0.877 ± 0.011** | **0.719 ± 0.015** | **0.756 ± 0.019** | **0.584 ± 0.028** | **0.734** |

*Bold marks best within each backbone block. ATAS leads every reported aggregate on MiniMax-H3, Wan 3.0, and HappyHorse 1.0, and remains ahead on ST-Bench and both ViStory measures for Wan 2.7.*

### Spatial Revisit Results

**Table 2: Spatial revisit** (target-image comparison under common-visibility mask)

| Method | PSNR$_C$ | SSIM$_C$ | LPIPS$_C$ |
|--------|----------|----------|-----------|
| I2V Extension | 13.601 | 0.293 | 0.339 |
| VGoT | 14.444 | 0.313 | 0.300 |
| Mora | 13.796 | 0.318 | 0.288 |
| MovieAgent | 14.543 | 0.352 | 0.296 |
| ReCA | 13.870 | 0.268 | 0.306 |
| FlexWorld | 14.740 | 0.343 | 0.315 |
| ViewCrafter | 15.255 | 0.375 | 0.293 |
| **ATAS** | **15.968** | **0.385** | **0.284** |

### Event Recapture Results

**Table 3: Source-grounded event recapture** (VLM judge, five criteria)

| Method | Event id. | Action state | Temporal order | Char. object | View realization |
|--------|-----------|--------------|----------------|--------------|------------------|
| I2V Extension | 0.450 | 0.433 | 0.433 | 0.817 | 0.200 |
| VGoT | 0.517 | 0.533 | 0.400 | 0.850 | 0.400 |
| Mora | 0.286 | 0.339 | 0.357 | 0.768 | 0.250 |
| MovieAgent | 0.500 | 0.450 | 0.500 | 0.783 | 0.350 |
| ReCA | 0.733 | 0.700 | 0.617 | 0.917 | 0.500 |
| **ATAS** | **0.750** | 0.688 | **0.688** | **0.917** | **0.650** |

ATAS leads event identity, temporal order, and viewpoint realization, ties ReCA on character/object consistency, and remains slightly lower on action-state preservation (0.688 vs. 0.700).

### Human Evaluation

ATAS receives the highest mean rating on **all six criteria**:
- Script faithfulness: **4.40**
- Character consistency: **4.22**
- Visual appeal: **4.10**
- Narrative coherence: **3.77**
- Background consistency: **3.62**

### Ablation Studies

**Table 4: Agent-component ablation** (cumulative components, Wan 3.0)

| Configuration | ViStory avg ↑ | C-Top10 ↑ | LPIPS$_C$ ↓ | Event Recapture ↑ |
|---------------|---------------|-----------|-------------|-------------------|
| Shot rendering only | 0.478 | 0.528 | 0.341 | 0.441 |
| + Scene/event resolution | 0.612 | 0.568 | 0.334 | 0.508 |
| + Recursive compilation | 0.638 | 0.684 | 0.288 | 0.711 |
| + Observation validation | 0.651 | 0.702 | 0.286 | 0.726 |
| + State update check | 0.670 | 0.719 | 0.284 | 0.739 |

**Table 5: Memory analysis** (varying stored evidence, Wan 3.0)

| Configuration | C-Top10 ↑ | LPIPS$_C$ ↓ | Recapture ↑ |
|---------------|-----------|-------------|-------------|
| No memory | 0.536 | 0.341 | 0.452 |
| Frame history | 0.598 | 0.319 | 0.548 |
| Static scene | 0.617 | 0.289 | 0.583 |
| Updated scene | 0.658 | 0.286 | 0.585 |
| Scene + event | 0.703 | 0.285 | 0.728 |
| Full state | 0.719 | 0.284 | 0.739 |

### Efficiency Analysis

- **Retries fall** from 1.00 to 0.54 per shot
- **Retry-adjusted delivery time declines** from 359 to 339 seconds per shot
- **Per-story runtime rises** from 51.9 to 64.9 minutes
- **Memory overhead**: 98 seconds per shot
- **Throughput falls** from 7.28 to 6.21 shots per hour
- **Film-review rejection decreases** from 52.6% (no memory) to 36.8% (full state)

## Theoretical and Practical Implications

### Theoretical Contributions

1. **World-state-centric paradigm**: ATAS reframes agentic video generation as a persistent-state management problem rather than a prompt-chaining problem. The formal separation of scene memory (static geometry) from event memory (dynamic observations) provides a principled decomposition of what must be remembered.

2. **Validation-controlled state evolution**: The separation of film acceptance ($f_t$) from state admission ($a_t$) is a novel theoretical contribution. Equation 11 formalizes the decision rule, preventing unreliable generated evidence from contaminating future generations while still allowing clips to be delivered as film.

3. **Addressability in space and time**: The framework demonstrates that explicit spatial anchors ($\xi_t$) and event-time intervals ($\tau_t$) make past observations queryable in ways that implicit context (recent frames, text summaries) cannot achieve.

### Practical Implications

- **Interactive filmmaking**: Directors can request return shots or event replays from new camera angles without regenerating from scratch.
- **Episodic content creation**: Long-running series can maintain consistent environments and event histories across episodes.
- **Previsualization**: Filmmakers can explore alternative camera angles on established scenes and events.
- **Error containment**: The validation mechanism prevents a single bad generation from cascading through the entire film.

### Key Findings

- The improvement pattern across open-source and proprietary backbones suggests gains come from **persistent multi-shot state** rather than any single renderer.
- Scene and event memory contribute **complementary** improvements: scene memory primarily improves spatial consistency (LPIPS$_C$), while event memory gives the largest recapture gain.
- Recursive compilation amplifies the benefits of memory, suggesting that **request decomposition** is crucial for leveraging persistent state.
- The remaining difficulty is **exact action-state reproduction** rather than retrieval of the requested occurrence or view.

## Conclusion

ATAS compiles screenplays into shot requests grounded in explicit scene, camera, and event-time references, then updates persistent state only from validated observations. The framework demonstrates:

- **Improved cross-shot consistency** across four video generators
- **Superior spatial revisit** performance compared to agent-based and 3D-guided methods
- **Strong event recapture** capabilities with a source-grounded evaluation protocol
- **Highest human ratings** on all six evaluation criteria
- **Reduced retry demand** at a measurable compute cost

The ablations show complementary contributions from scene memory, event memory, and recursive compilation. By keeping locations and prior events addressable throughout a story, ATAS provides a foundation for agentic video tools in interactive filmmaking, episodic content creation, and previsualization.

**Future directions** implied by the work include:
- Improving exact action-state reproduction in event recapture
- Reducing computational overhead of memory maintenance
- Extending the framework to handle more complex multi-scene interactions
- Addressing environment-wide consistency and whole-story coherence, which received lower human ratings

---

_Markdown view of https://picx.dev/p/3RYm0O, served by PicX — AI-generated visual whiteboard summaries of research papers._