Advances in Agentic Computer Graphics

Authors: Hengyu Liu, Jihyeon Je, Ruiqi Wang, Mingxuan Wu, Haven Feng, Qixing Huang, Georgios Pavlakos, Hao Zhang, Jiajun Wu, Maneesh Agrawala, Leonidas Guibas, R. Kenny Jones, Shangzhe Wu, Guandao Yang

Affiliations: UT Austin, Stanford University, Simon Fraser University, UC Berkeley, University of Cambridge

---

Summary (Overview)

• This paper presents a comprehensive survey of agentic graphics systems—systems that use large language models (LLMs) to perform computer graphics tasks through iterative execution, feedback, and refinement.
• The authors introduce a unified four-dimensional taxonomy (specification, base model, harness, graphics interface & tooling) to characterize agentic graphics systems across diverse applications, addressing the challenge of inconsistent terminology across the literature.
• They develop an agentic literature survey system that automatically discovers, reads, annotates, and curates papers using the taxonomy, compiling a corpus of 350 papers published through September 2026.
• The survey analyzes 2D graphics, static 3D assets, and dynamic 3D assets, revealing key design patterns: visual feedback dominates (77.0% of works), local patching is increasingly common (rising from 50.0% to 83.0%), and documentation/rules are becoming essential resources (85.3% in latest period).
• The paper identifies open challenges including multi-agent collaboration, tool orchestration, agent-oriented interfaces, long-horizon planning, transparency, human intent alignment, reliable evaluation, and creativity/attribution.

---

Introduction and Theoretical Foundation

Background and Motivation

Computer graphics is fundamentally built on code—executable procedures that create and manipulate visual content. Recent advances in LLMs, particularly in code generation, reasoning, and tool use, have opened new opportunities for automating complex graphics workflows. Agentic graphics systems leverage LLM agents that interact with graphics environments, execute operations, inspect results, and use feedback to guide subsequent actions.

The field faces two critical challenges:

1. Scale: The corpus exceeds 300 papers published through September 2026, growing at an exponential rate
2. Terminology fragmentation: Functionally similar components are described with different names across applications (e.g., feedback modules, planning modules), obscuring shared design choices

Definition of Agentic Graphics

An agentic graphics system must satisfy three criteria:

1. An LLM makes at least some action decisions
2. The system executes operations in a graphics environment to create or modify graphics content
3. Feedback on intermediate results or execution is used to select or revise subsequent actions

This definition is intentionally inclusive—imposing no restrictions on graphics representation, software platform, training strategy, or automation level. It explicitly excludes:

• Networks trained from scratch to directly solve graphics tasks
• Single-pass LLM generation without subsequent feedback
• Methods using feedback only for training or final evaluation
• Multi-step LLM pipelines where environment feedback doesn't influence model actions

Theoretical Foundations

The work builds on several established areas:

• Traditional graphics workflows: Procedural rules, shading programs, numerical solvers [MWH*06; STBB14; Coo84]
• Program generation: Shape2Prog [TLS19], ShapeAssembly [JBX20] learn to infer programs from shapes
• Language model agents: LLMs select actions based on task instructions and observations from external environments [XCG*23]
• Neuro-symbolic modeling: Inference of graphics programs from observations [RGJ*23]

---

Methodology

The Unified Taxonomy

The paper proposes a four-dimensional design space (Figure 3):

1. Specification

• Task: 2D Graphics, Static 3D Graphics, Dynamic 3D Graphics
• Input Form: Language, Image, Video, Existing 3D Representation, Document
• Output: Program, Structured Specification, Graphics Artifact, Rendered Media

2. Base Model

• Model Type: LLM, Vision-Language Model (VLM), Hybrid systems
• Model Training: Off-the-shelf, Supervised Fine-tuning, Reinforcement Learning post-training

3. Harness

• Plan & Act:
  • Planning: Hard-Coded, Hybrid, Non Hard-Coded
  • Workflow: Sequential, Parallel, Graph-structured
  • Update Strategy: Regeneration, Local Patch, Rollback, Replanning, Candidate Selection
• Feedback Mechanism:
  • Verifier: Self-verification, Critic, Programmatic, Environment-based, Human
  • Feedback Signal: Binary verdict, Score, Ranking, Diagnosis, Visual observations, Structural state, Runtime observations
• Memory:
  • Time: Working, Short-term, Long-term
  • Reach: Local, Global

4. Graphics Interface & Tooling

• Action Type: Code/Scripts, Structured Specification, Tool/Function Call, GUI Action
• Graphics Software: Agent-native, Agent-adapted, General-purpose
• Tools: Graphics Operations, Perception Models, Validators
• Resources: Assets, Examples & Trajectories, Documentation & Rules
• Representation: Code, Geometry, Dynamic State, Scene Structure/Layouts, Pixels, Learned State

Agentic Literature Survey System

The authors built an automated system (Figure 4) with three stages:

1. Paper Discovery: Scans publication listings, searches the web with customized queries (iteratively refined via Re-Act framework [YZY*23]), merges related works, removes duplicates, verifies publication versions

2. Evidence-based Annotation: Reads text, inspects figures/tables using dedicated tools [YJW24], gathers supporting evidence (following PaperQA2 [SCL24]), assigns taxonomy labels with source references, distinguishes reported facts from interpretations

3. Knowledge Curation: Maintains curated records independent of working context [SYNG24], verifies required fields and taxonomy adherence, flags insufficient evidence for revisiting, saves papers independently for interruption recovery

Key implementation details: The system uses GPT-6 Astra with medium reasoning effort, follows SWE-agent [YJW*24] for research tools, and is planned for public release.

---

Empirical Validation / Results

Corpus Statistics

• 350 papers compiled between 2024 and September 2026, followed by manual verification of relevance

Key Findings from Corpus Analysis (Figure 5)

(1) Feedback Observations

• Visual feedback is the most widely used observation type: 77.0% of 304 works using observation as feedback
• 58.2% combine multiple observation categories, suggesting single observation types are often insufficient
• Task variation: Scene generation dominates visual/structural observations; CAD generation dominates runtime observations (reflecting emphasis on geometric precision and executable correctness)

(2) Update Strategies

• CAD generation more frequently uses full regeneration
• Scene generation more frequently uses local patch updates (scene representations allow independent object modification, while CAD operations have interdependent modeling steps)
• Some systems evaluate whether revisions should be retained (e.g., SceneSmith restores checkpoints when critic scores decline)

(3) Supporting Resources

• Scene generation most frequently uses asset libraries (reusable scene components)
• CAD generation relies on documentation and rules (for selecting appropriate modeling operations)

(4) Temporal Trends

• Local patching rising from 50.0% to 83.0% across the analyzed period
• Documentation and rules reaching 85.3% of records in the latest period
• Trend toward systems integrating richer feedback, making targeted revisions, and relying more heavily on explicit task knowledge

Example Annotations (Table 1)

Benchmarks and Evaluation

The survey covers benchmarks across three levels:

1. Spatial understanding: General benchmarks for spatial relationships, distances, 3D grounding [CXK24; HZC23; XWW*24]

2. Graphics artifact evaluation:

   • CAD-Assistant [MKC25], 3DCodeBench [GSY26], Voxel-CodeBench [ZB26], BrickBench [KXJ*26]
   • MoVer [MA25] for motion graphics with first-order spatio-temporal logic predicates
   • IR3D-Bench [LLL*25] for image-to-3D reconstruction
   • BlenderGym [GHJ25], Op-CAD [BGX26], VIGA [YGW26], BVB [TSM26], 4DCodeBench [SKK26] for multi-step workflows

3. Agentic interaction and coordination:

   • ALE [SHZ26], OSWorld [XZC24] for computer-use agents
   • 3DHarnessBench [LGPO26], SceneActBench [ZZY*26] for interactive reconstruction
   • CADWorld [DLM*26] for CAD workflows via GUI

---

Theoretical and Practical Implications

Design Pattern Insights

The taxonomy reveals recurring patterns across applications:

1. Feedback grounding: Visual feedback dominates due to VLM advances, but is often insufficient alone—geometric measurements and simulations provide complementary structural/physical validity information

2. Update strategy-task fit: The choice between regeneration and local patching correlates with representation editability—scenes with independently modifiable objects favor local patches, while CAD's interdependent operations favor regeneration

3. Resource-task alignment: Scene generation leverages asset libraries while CAD relies on documentation, reflecting different task requirements (arrangement vs. precise tool use)

4. Trend toward targeted revision: The shift from regeneration to local patching (50%→83%) suggests growing sophistication in preserving successful work while fixing errors

Application-Specific Findings

2D Graphics (Sec. 6): Five categories—SVG/vector graphics, visual design/editing, scientific figures/diagrams, data visualization, document/presentation design. Key insight: grounding structured visual representations in rendered evidence while preserving component-level editability.

Static 3D Assets (Sec. 7):

• Object modeling: Geometry construction, texture/material modeling, asset editing—central challenge is translating descriptions into geometry reflecting intended part structure
• CAD modeling: Specification-driven generation, reference-based reconstruction, model editing—geometric validity and design conformance are central
• Scene modeling: Generation, reconstruction, editing—spatial coherence and requirement conformance are key

Dynamic Assets (Sec. 8): Articulation, animation, simulation—requires feedback connecting intended behavior to responsible structures/controls. Key distinction: matching visible motion ≠ recovering the physical mechanism producing it (4DCodeBench demonstrates strong static reconstruction doesn't ensure accurate temporal behavior)

Open Problems (Sec. 9)

1. Multi-agent collaboration and cross-task generalization: Moving from task-specific systems to general frameworks; sharing knowledge across tools/domains
2. Action spaces and tool orchestration: Determining appropriate actions/tools per task; balancing reliability, editability, and cost
3. Agent-oriented interfaces and representations: Developing languages/abstractions optimized for agent reasoning (e.g., AIDL [JZH*25] with hierarchical solver-aided CAD language); bridging machine-oriented and human-understandable representations
4. Long-horizon planning, memory, and efficiency: Maintaining project state, tracking decisions/dependencies, learning from failures, reusing strategies
5. Transparency and interpretability: Exposing plans, assumptions, intermediate artifacts, edit histories without overwhelming users
6. Human interaction and intent alignment: Handling underspecified instructions, proactive clarification, explicit assumption management, fine-grained feedback
7. Reliable evaluation and validation: VLMs overlook subtle errors, don't align with human perception; stochastic workflows complicate evaluation; disentangling system design from underlying model improvements
8. Creativity and attribution: Examining how creative decisions are distributed between users and agents; authorship, provenance, accountability

---

Conclusion

This survey establishes agentic graphics as a class of systems where a base model performs graphics operations through an agentic harness in a graphics interface. Key takeaways:

• Recurring design patterns exist across tasks in planning, verification, and iterative refinement
• Most systems use off-the-shelf base models—limited exploration of supervised or RL post-training tailored to graphics
• Harness designs show greater diversity than base model choices, reflecting different orchestration strategies
• Code and structured specifications dominate interaction forms, with tool calls and GUI actions adopted based on available interfaces
• Evaluation is expanding beyond final artifact quality to multi-step editing, interaction trajectories, and workflow effectiveness

The taxonomy is envisioned as an adaptive framework—its high-level structure accommodates future developments while individual dimensions can be expanded as new approaches emerge. The authors hope this survey provides a common vocabulary for understanding existing approaches, identifying trends, and guiding development of more capable, reliable, and generalizable agentic graphics systems.