Summary (Overview)

• Unify-Agent is the first end-to-end unified multimodal agent for world-grounded image synthesis, reformulating text-to-image generation as an agentic pipeline of Think → Research → Recaption → Generate rather than a single-pass mapping.
• The model is built upon Bagel, a Mixture-of-Transformers (MoT) unified multimodal model, and is fine-tuned on 143K curated agent trajectories covering 12 categories of long-tail, knowledge-intensive concepts (Celebrity, Animation, Game, Comic, Mythology, etc.).
• The authors introduce FactIP, a new benchmark with 2,462 curated prompts (500 test split) specifically designed to evaluate factual fidelity and identity consistency for rare and culturally significant concepts.
• Unify-Agent achieves an Overall score of 73.2 on FactIP, surpassing its base model (Bagel, 50.9) by more than 22 points and outperforming strong generation-only baselines like FLUX.1-dev (28.9) and SD-3.5-large (27.5).
• The work demonstrates a key architectural insight: in unified multimodal models, generative priors (VAE latents) enhance multimodal understanding, enabling superior evidence recaptioning compared to decoupled pipelines.

---

Introduction and Theoretical Foundation

Background and Motivation

Recent advances in Text-to-Image (T2I) generation have significantly improved visual realism, controllability, and stylistic diversity. However, a growing challenge is whether models can faithfully depict entities grounded in the real world—including real people, cultural symbols, rare intellectual properties (IPs), historical scenes, and scientific phenomena. Success requires not just visually plausible outputs but factual and visual fidelity to the intended entity.

Unified multimodal models (UMMs) offer a promising approach by unifying visual understanding and image generation within a shared architecture. However, existing models rely primarily on frozen parametric knowledge, which is often insufficient to recover the correct appearance and identity-defining visual cues of long-tail concepts. This reflects a fundamental constraint: world knowledge embedded in fixed model parameters cannot be updated at inference time.

Key Insight: From Closed-Book to Open-Book Generation

The authors argue that addressing this bottleneck requires moving beyond closed-book generation toward open-book, agentic generation, where models access external world knowledge at inference time. Existing agentic T2I systems fall short because they are brittle, multi-stage pipelines that loosely connect LLM planners, retrieval tools, and standalone generators. The key challenge is not simply retrieving external knowledge, but turning it into visual guidance that preserves fidelity to source knowledge while remaining aligned with user intent.

Theoretical Foundation: Problem Formulation

Standard T2I models the conditional distribution $p_\theta(y|x)$, where an image $y \in \mathcal{V}$ is synthesized from a user prompt $x \in \mathcal{X}$ relying strictly on parametric memory $\theta$. For world-grounded synthesis involving rare entities, this is intractable due to knowledge deficits in $\theta$.

The authors formulate world-grounded synthesis as an interleaved generative trajectory over an augmented state space with four intermediate variables: cognitive gap assessment $g$, textual evidence trace $\tau_t$, visual evidence trace $\tau_v$, and evidence-grounded recaption $c$:

$$p_\theta(y, c, \tau_t, \tau_v, g \mid x) = \underbrace{p_\theta(g \mid x)}_{\text{Gap Detection}} \cdot \underbrace{p_\theta(\tau_t, \tau_v \mid x, g)}_{\text{Evidence Acquisition}} \cdot \underbrace{p_\theta(c \mid x, g, \tau_t, \tau_v)}_{\text{Evidence-Grounded Recaptioning}} \cdot \underbrace{p_\theta(y \mid c, \tau_v)}_{\text{Visual Synthesis}} \tag{3}$$

This factorization defines four cognitive phases: (1) detecting knowledge gaps, (2) acquiring multimodal evidence, (3) recaptioning evidence into structured constraints, and (4) evidence-grounded visual synthesis.

---

Methodology

3.1 Base Architecture: Bagel

Unify-Agent is built upon Bagel, which employs a Mixture-of-Transformers (MoT) architecture integrating a ViT encoder. The model disentangles two core capabilities:

Multimodal Understanding (autoregressive next-token prediction):

$$\mathcal{L}_{\mathrm{text}} = -\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t}, C) \tag{1}$$

Multimodal Generation (rectified flow in VAE latent space):

$$\mathcal{L}_{\mathrm{image}} = \mathbb{E}_{t \sim \mathcal{U}(0,1), z_t} \left\| u_\theta(z_t, t; C) - u^\star(z_t, t) \right\|_2^2 \tag{2}$$

3.2 Motivating Evidence

A preliminary training-free study on 200 examples from FactIP categories (scene, character, object) compared four inference settings: (1) prompt-only, (2) text injection, (3) visual injection, and (4) text+visual injection. Results showed:

• Both textual and visual knowledge improve over prompt-only baseline
• Visual injection yields substantially larger gains than text injection
• Naive text+visual injection is slightly weaker than visual alone, motivating the recaption paradigm

4. Data Pipeline

Training Data Construction — Each training sample is represented as:

$$\mathcal{D}_{\mathrm{SFT}} = \{(x, \tau_t, \tau_v, c)\} \tag{4}$$

The pipeline has three stages:

1. Task Source and Prompt Collection: 456K curated IPs across 12 domains, with structured metadata from GPT-4o and manual verification
2. Multimodal Research Trace Construction: Using Claude Opus 4.6 as teacher agent, constructing textual research traces $\tau_t = (q_t, E_t)$ and visual research traces $\tau_v = (q_v, E_v)$, where visual candidates are scored via:

$$s(v_i) = \sum_{k=1}^{4} \lambda_k s_k(v_i \mid x, E_t) \tag{9}$$

and top-2 images are selected as visual evidence:

$$E_v = \operatorname{arg\,top2}_{v_i \in \widetilde{E}_v} s(v_i) \tag{10}$$

3. Evidence-Grounded Recaption Annotation: Using Nano Banana Pro for generation-based validation with reject-sampling (up to 5 trials), yielding 143K high-quality trajectory-image pairs

FactIP Benchmark — 2,500 samples through manual filtering, with a 500-sample test split. Evaluated on four dimensions: Clarity, Content, Aesthetics, and Relevance.

5. Fine-Tuning and Inference

Training Objective (dual-loss design):

$$\mathcal{L}_{\mathrm{SFT}} = \mathcal{L}_{\mathrm{text}} + \mathcal{L}_{\mathrm{image}} \tag{14}$$

Text supervision uses autoregressive next-token prediction with upweighted special tokens (thinking, tool_call, recaption tags). Image supervision uses latent flow-matching. A hybrid attention masking strategy regulates information flow across reasoning traces, reference images, recaption tokens, and generation tokens.

Inference Pipeline (Think → Research → Recaption → Generate):

• Think: Structured prompt understanding with cognitive gap detection $g \sim p_\theta(g|x)$, representing missing knowledge as units $\mathcal{M}(x) = \{m_1, \dots, m_K\}$
• Research: Sequential textual-then-visual evidence acquisition
• Recaption: Transform evidence into identity-preserving and scene-compositional constraints, then integrate into recaption $c \sim p_\theta(c|x, g, \tau_t, \tau_v)$
• Generate: Final synthesis conditioned solely on recaption and visual anchors: $y \sim p_\theta(y|c, \tau_v)$

---

Empirical Validation / Results

FactIP Benchmark Results

Unify-Agent achieves the highest Overall score of 73.2 among Unified MLLMs, with exceptional performance on the Relevance dimension (Character: 67.3, Object: 71.8, Scene: 78.2). It vastly surpasses generation-only models like FLUX.1-dev (28.9) and SD-3.5-large (27.5).

WiSE Benchmark

Unify-Agent attains the best Overall score of 0.77 within the Unified MLLM category, exceeding BAGEL+CoT (0.70) and excelling in cultural (0.82), biological (0.72), and chemistry (0.70) knowledge.

KiTTEN Benchmark

Unify-Agent establishes a new state of the art with an overall score of 4.08, outperforming strong baselines like Imagen-3 (3.50), with highest text alignment (4.22) and entity alignment (3.93).

T2I-FactBench

Within the unified MLLM group, Unify-Agent achieves top scores in SKCI (77.4) and MKCC (71.5), with SKCM concept score of 69.2 comparing favorably against commercial models like DALLE-3 (55.5).

Ablation Study

Key findings:

• Removing image search causes the largest degradation (Overall 56.2), especially on Relevance (72.4 → 50.8)
• Removing ViT (Recaption w/o ViT) is more damaging than removing VAE, confirming that high-level semantic tokens are critical for understanding
• The full model improves Relevance from 44.9 to 72.4, showing the main advantage lies in factual grounding

---

Theoretical and Practical Implications

Generation Helps Understanding in Unified Models

A central insight is that generation can improve understanding in unified understanding-generation models. The recaption stage requires transforming retrieved reference images into structured textual specifications—a multimodal understanding problem requiring identification of identity-critical attributes. In Bagel-style architectures, visual inputs are tokenized through a dual VAE+ViT design:

• ViT captures high-level semantics (global context, entity identity)
• VAE provides low-level perceptual latents (texture, material, structural geometry)

Together, these enable superior multimodal reasoning during evidence recaptioning, demonstrating a mutually reinforcing synergy between understanding and generation capabilities.

Practical Implications

1. Agentic Paradigm for T2I: Reframing generation as a sequential decision process (Think, Research, Recaption, Generate) enables reliable synthesis of rare, long-tail, and knowledge-intensive concepts that closed-book models cannot handle.
2. Evidence-Grounded Recaptioning: The recaption serves as a structured interface that filters noisy raw evidence and reorganizes it into generation-compatible constraints, resolving the modality conflict of naive injection.
3. Unified Architecture Advantage: End-to-end unified models eliminate the cascading errors of brittle multi-API pipelines, enabling genuine reasoning-driven generation.

---

Conclusion

Unify-Agent pioneers agentic modeling for world-grounded image synthesis, demonstrating that tightly coupling reasoning, searching, and generation enables reliable open-world image synthesis. Key contributions include:

1. A Novel Agentic Paradigm: The first end-to-end unified multimodal agent reformulating T2I as an active, inference-time sequential decision process
2. Architectural Insights: Revealing that unifying understanding and generation creates mutually reinforcing synergy, where generative latents enhance multimodal reasoning
3. Comprehensive Benchmark: FactIP, a curated benchmark for evaluating identity consistency and factual faithfulness of rare and long-tail concepts
4. Superior Performance: Setting new records among open-source unified models across FactIP, WiSE, KiTTEN, and T2I-FactualBench, approaching the world knowledge capabilities of leading commercial models

Future Directions: The work highlights the value of open-book, evidence-grounded generation and suggests that further improvements could come from expanding the agentic pipeline to more complex multi-step reasoning, broader knowledge sources, and more sophisticated evidence integration strategies.