# ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

> ToolArtist post-trains a unified multimodal model so that tool use and native image generation become actions of a single agent policy, outperforming fixed-pipeline and search-only baselines on open-world benchmarks.

- **Source:** [arXiv](https://arxiv.org/abs/2608.04436)
- **Published:** 2026-08-07
- **Permalink:** https://picx.dev/p/z0MUbK
- **Whiteboard:** https://picx.dev/p/z0MUbK/image

## Summary

## Summary (Overview)

- **ToolArtist** is a fully agentic image-generation model built by post-training a Unified Multimodal Model (UMM, specifically Emu3.5). It unifies autonomous reasoning, external tool invocation (text/image search), and **native** image generation under a **single policy**, rather than delegating synthesis to a separate generator or following a fixed pipeline.
- The training pipeline uses (1) **Supervised Fine-Tuning (SFT)** on 7,132 high-quality synthetic trajectories (collected via a teacher agent and converted into native multimodal format), and (2) a novel **Reason–Act–Draw GRPO (RAD-GRPO)** reinforcement learning algorithm that jointly optimizes both the generation *intent* (caption) and the generated *image* quality using complementary rewards.
- Extensive experiments on **WISE** and **WorldGenBench-Humanities** show that placing the *entire* open-world generation process under agent control outperforms existing approaches with only partial agentic capabilities (e.g., fixed-pipeline methods and prompt-optimization search agents).
- ToolArtist achieves the best non-proprietary average Knowledge Checklist Score on WorldGenBench-Humanities (22.10) and strong WISE overall scores (0.79), while remaining competitive with frontier proprietary models on knowledge-heavy natural science categories.
- The authors release the training data, code, and complete SFT/RL infrastructure publicly.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Text-to-image (T2I) models produce visually compelling images but fail on **open-world image generation** tasks that require:
- Complex semantic understanding,
- Multi-hop reasoning,
- Long-tail / time-sensitive world knowledge,
- Integration of external information that is not explicitly in the prompt or stored in the model’s static parameters.

Recent benchmarks (e.g., WISE, WorldGenBench) highlight this capability gap. Existing attempts to add agentic abilities either:
1. **Prescribe a fixed pipeline** (e.g., Unify-Agent) where search, evidence aggregation, and synthesis are ordered in advance, or
2. **Delegate generation to an external generator** after a learned search agent rewrites the prompt (e.g., GenSearcher).

Both approaches fail to let the model freely decide *when* and *how* to generate. The core thesis:

> Fully agentic image generation requires tool use and image generation to be autonomous actions of the same policy.

ToolArtist addresses this by making image generation a **native action** of the agent policy.

### Theoretical Formulation

**Open-world image generation.** Define the world-knowledge space as $\mathcal{W}$ and the task-relevant evidence as $\mathcal{Z} \subseteq \mathcal{W}$. Instead of assuming all information is available in the prompt $q$ or model parameters, the target image must be actively gathered:

$$
\mathcal{I} \sim G(\cdot \mid q, \mathcal{Z}), \qquad \mathcal{Z} \subseteq \mathcal{W}.
$$

**Unified Multimodal Model (UMM).** ToolArtist is built on Emu3.5, which models text and images as tokens in a unified autoregressive framework:

$$
p_\theta(\mathbf{x}) = \prod_{j=1}^{L} p_\theta(x_j \mid x_{<j}),
$$

where $x_j$ can be textual or visual tokens. This lets the model process multimodal observations and natively generate images within the same context.

**Agentic image generation.** Following ReAct-style interaction with the environment, the agent is defined as:

$$
A_\theta = (\pi_\theta, \mathcal{T}, \mathcal{W}),
$$

where $\pi_\theta$ is the UMM, and $\mathcal{T} = \{ \text{TextSearch}, \text{ImageSearch} \}$ is the tool set. At each round $t$, the policy reasons and selects an action:

$$
(r_t, a_t) \sim \pi_\theta(\cdot \mid \mathcal{H}_{t-1}), \qquad a_t \in \mathcal{A}_{\text{tools}} \cup \mathcal{A}_{\text{draw}}.
$$

- **Tool calling:** $a_t = (n_t, u_t)$, with tool name $n_t$ and query $u_t$; observations $o_t \sim \mathcal{W}(\cdot \mid a_t)$ (returned text, images, summaries) are appended to history.
- **Native image generation:** $a_t = (g_t, v_t)$, where the visual-caption span $g_t$ consolidates the user request and evidence into an executable generation intent, and $v_t$ is the generated visual-token span. Both are produced by the UMM itself.

The complete trajectory is:

$$
\mathcal{H}_T = [q, h_1, h_2, \dots, h_T],
\tag{1}
$$

with

$$
h_t = \begin{cases}
[ r_t, a_t, o_t ] & a_t \in \mathcal{A}_{\text{tools}}, \\
[ r_t, a_t ] & a_t \in \mathcal{A}_{\text{draw}}.
\end{cases}
$$

**Agentic masking.** To train the model, only policy-generated tokens (reasoning spans $r_t$ and action spans $a_t$) are supervised; user instructions and environment observations remain as context. The policy-support mask is $M_j = \mathbb{I}[y_j \in \mathbf{T}]$, where $\mathbf{T} = \bigcup_t (r_t \cup a_t)$.

---

## Methodology

### Stage 1: SFT — Data Synthesis and Conversion

**Teacher rollouts.** A teacher agent (using Gemini-3-Pro-Image-Preview for actual image generation) interacts with open-world tools (Google Search + LLM Reader for text; filtered image search with source-aware summaries) to produce **raw multi-turn trajectories**: reasoning → tool calls → evidence → final image generation. The rollout is not a fixed “search-then-draw” order; it is fully flexible.

**Conversion.** The raw trajectories are converted into UMM-compatible format:
- Search tool outputs (text + images) remain in context.
- The external image-generation tool is **concealed**: the prompt used to call it is rewritten as a **visual-caption span** followed by the **image tokens**, so the UMM internally performs generation.
- Trajectories that fail before the final image, or fail loading/tokenization/context-length checks, are filtered out.

The final SFT dataset contains **7,132 trajectories**. The SFT loss is:

$$
\mathcal{L}_{\text{SFT}}(\theta) = - \mathbb{E}_{(x_i, y_i) \sim \mathcal{D}_{\text{SFT}}}
\left[
\frac{\sum_{j=1}^{L_i} M_{i,j} \log P_\theta(u_{i,j} \mid u_{i,<j})}
{\sum_{j=1}^{L_i} M_{i,j}}
\right],
$$

where $M_{i,j}$ is the agentic mask indicator. This supervises the complete multimodal policy trajectory — reasoning, searching, evidence utilization, caption generation, and native image token production — while excluding conditioning prefix and environment observations.

### Stage 2: RL — Reason–Act–Draw GRPO (RAD-GRPO)

RAD-GRPO directly optimizes full inference trajectories online. For each prompt $q$, the old policy samples a group of $B$ trajectories interacting with the environment:

$$
\mathcal{H}_i \sim \pi_{\theta_{\text{old}}}(\cdot \mid q) \otimes \mathcal{W}, \qquad i = 1, \dots, B.
$$

**Dual reward.** Each trajectory $\mathcal{H}_i$ with termination step $T_i$ has a final generation action $(g_{T_i}, v_{T_i})$:

- **Intent reward** $R_i^{I}$: evaluates whether the final caption $g_{T_i}$ is a sufficient, accurate, and executable translation of the user request and evidence — i.e., whether an ideal generator would produce the right image from it.
- **Quality reward** $R_i^{Q}$: evaluates the final image decoded from $v_{T_i}$ across four dimensions — faithfulness (0.1), visual correctness (0.4), text accuracy (0.4, falling back to 0

---

_Markdown view of https://picx.dev/p/z0MUbK, served by PicX — AI-generated visual whiteboard summaries of research papers._