# Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning

> The paper introduces process-driven image generation, a multi-step method that interleaves textual reasoning and visual actions to improve compositional alignment and world knowledge reasoning in image synthesis.

- **Source:** [arXiv](https://arxiv.org/abs/2604.04746)
- **Published:** 2026-04-10
- **Permalink:** https://picx.dev/p/LiWyrW
- **Whiteboard:** https://picx.dev/p/LiWyrW/image

## Summary

# Summary of "Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning"

## Summary (Overview)
*   **Paradigm Shift:** Introduces **process-driven image generation**, a multi-step paradigm that decomposes image synthesis into an interleaved trajectory of textual reasoning and visual actions, moving beyond single-pass, black-box generation.
*   **Core Process:** Employs a recurring four-stage cycle—**Plan, Sketch, Inspect, Refine**—where textual reasoning explicitly conditions visual evolution, and generated visual intermediates ground the next round of textual reasoning.
*   **Key Innovations:** Solves the challenge of supervising ambiguous intermediate states via **scene-graph subsampling** for logical trajectories, **self-sampled critique data** for teaching error detection, and **end-to-end training** of a unified multimodal model to autoregressively generate interleaved tokens.
*   **Performance Gains:** Lifts the base BAGEL-7B model from **79% to 83% (+4%)** on the GenEval benchmark for compositional alignment and from **70% to 76% (+6%)** on the WISE benchmark for world knowledge reasoning.
*   **Efficiency:** Achieves superior performance with significantly lower training data (62K vs. 688K samples) and inference cost (131 vs. 1000 sampling steps) compared to prior process-level approaches like PARM.

## Introduction and Theoretical Foundation
Current text-to-image models often produce plausible but incorrect images due to the challenge of resolving complex spatial layouts, object relations, and fine-grained attributes in a single forward pass. While textual chain-of-thought (CoT) reasoning has been explored, it remains "visually blind," unable to perceive or correct spatial misalignments during generation. Existing multimodal CoT or tool-augmented approaches often decouple reasoning from generation or apply it as a post-hoc repair.

This paper challenges the outcome-driven paradigm by proposing **process-driven image generation**, which reformulates generation as a **co-evolving trajectory of textual plans and visual states**. Inspired by how humans paint incrementally, the method aims to make the generation process explicit, interpretable, and directly supervisable. The core hypothesis is that a unified multimodal model can be trained to perform **interleaved reasoning**, where textual and visual modalities mutually inform and constrain each other throughout a multi-step construction process, enabling error correction as it emerges.

## Methodology

### Framework
The model performs generation as a sequential, interleaved textual-visual reasoning process. Given a unified multimodal model $P_\theta$ and an input text prompt $T$, it generates a trajectory of alternating textual reasoning steps $s^{(i)}$ and intermediate visual states $v^{(i)}$, ultimately converging to the final image $I$:

$$
\{ s^{(1)}, v^{(1)}, s^{(2)}, v^{(2)}, ..., v^{(k)}, s^{(k)} \}, I \sim P_\theta(\cdot|T) \tag{1}
$$

This is realized through a recurring four-stage cycle:
1.  **Plan:** The model generates an incremental instruction `<ins>` and a global scene description `<des>`.
2.  **Sketch:** Conditioned on the plan, the model synthesizes an updated draft image.
3.  **Inspect:** The model detects conflicts between (a) the textual plan/description and the original prompt, and (b) the sketch and the planned instruction.
4.  **Refine:** If discrepancies are found, the model emits a refinement instruction `<refine>` and generates a corrected visual update.

The overall pipeline is summarized as:
$$
T \rightarrow s^{(1)}_{plan} \rightarrow v^{(1)}_{sketch} \rightarrow s^{(1)}_{inspect} \rightarrow v^{(1)}_{refine} \rightarrow ... \rightarrow I \tag{2}
$$

### Intermediate Reasoning Collection
A multi-stage pipeline constructs high-quality, process-oriented reasoning traces for supervised fine-tuning.

*   **Multi-Turn Generation Subset:** Creates the base for step-by-step generation. Uses **scene-graph subsampling** to derive a sequence of incremental, logically ordered prompts that expand a scene without contradictions. Augmented with GPT rewriting to enrich editing actions (modify, swap, remove).
*   **Instruction-Intermediate Conflict Reasoning Subset:** Improves textual-side reasoning. Uses **self-sampling** from a fine-tuned model, with GPT as a judge to evaluate consistency with the original prompt and generate critiques for conflicts.
*   **Image–Instruction Alignment Reasoning Subset:** Improves visual-side reasoning. Extends the Gen-Ref dataset into positive (consistent) and negative (misaligned) samples, with GPT providing explanations and corrective instructions.

**Table 1: Statistics of intermediate reasoning dataset.**
| **Reasoning Subset** | **Total Samples** | **Details** |
| :--- | :--- | :--- |
| **Multi-turn Generation** | 32,012 | Avg. Prompt Length: 152.8, Avg. Images per Sample: 3.51 |
| **Instruction-Intermediate Conflict** | 15,201 | Positive: 6,905, Negative: 8,296 |
| **Image-Instruction Alignment** | 15,000 | Positive: 5,000, Negative: 10,000 |

### Model & Training
The backbone is a unified multimodal model (BAGEL-7B). It is trained to autoregressively generate interleaved sequences of text and image tokens.

*   **Text Loss:** Cross-Entropy (CE) loss applied to textual segments $s^{(i)}$ and special vision boundary tokens.
    $$
    \mathcal{L}^{\text{text}}_{\text{CE}} = - \sum_{t \in [1,i]} \log P_\theta\left(s_t | y_{<t}, T \right) \tag{3}
    $$
*   **Image Loss:** Mean Squared Error (MSE) loss based on the Rectified Flow paradigm.
    $$
    \mathbf{z}^{(i)}_t = t \cdot \mathbf{z}^{(i)}_0 + (1-t) \cdot \mathbf{z}^{(i)}_1, \quad t \in [0, 1] \tag{4}
    $$
    $$
    \mathcal{L}^{\text{image}}_{\text{MSE}} = \mathbb{E}\left[ \left( P_\theta\left(\mathbf{z}^{(i)}_t | y_{<t}, T \right) - (\mathbf{z}^{(i)}_0 - \mathbf{z}^{(i)}_1) \right)^2 \right] \tag{5}
    $$
*   **Total Loss:** A weighted combination.
    $$
    \mathcal{L}_{\text{total}} = \lambda_{\text{CE}} \cdot \mathcal{L}^{\text{text}}_{\text{CE}} + \mathcal{L}^{\text{image}}_{\text{MSE}} \tag{6}
    $$

## Empirical Validation / Results

### Quantitative Evaluation

**Table 2: Evaluation on GenEval benchmark (compositional alignment).**
| **Model Category** | **Model** | **Single Object** | **Two Objects** | **Position** | **Color Attr.** | **Overall** |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| **Generation Only** | FLUX.1-dev (12B) | 0.98 | 0.93 | 0.68 | 0.65 | **0.82** |
| **Unified Multimodal** | BAGEL-7B* | 0.99 | 0.95 | 0.51 | 0.56 | 0.77 |
| **Unified Multimodal** | **Ours (BAGEL-7B + Process-Driven)** | **0.99** | **0.95** | **0.72** | **0.69** | **0.83** |

*Our method boosts BAGEL-7B by 5% absolute gain, achieving comparable performance to the 12B FLUX.1-dev model.*

**Table   3: Evaluation on WISE benchmark (world knowledge reasoning).**
| **Model Category** | **Model** | **Culture** | **Time** | **Chemistry** | **Overall** |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Generation Only** | FLUX.1-dev | 0.48 | 0.58 | 0.35 | 0.50 |
| **Unified Multimodal** | BAGEL | 0.76 | 0.69 | 0.58 | 0.70 |
| **Unified Multimodal** | **Ours (BAGEL + Process-driven)** | **0.74** | **0.82** | **0.78** | **0.76** |

*Our method boosts BAGEL-7B by 8.5% absolute gain, with particularly large gains on challenging domains like Time and Chemistry.*

### Analysis of Process-driven Reasoning

**Table 4: Comparison with process-driven baselines on GenEval.**
| **Reasoning Strategy** | **Training Dataset** | **Inference Cost** | **Gen-Eval Score** |
| :--- | :--- | :--- | :--- |
| Training-free: BAGEL + GPT (Planner) | - | 50 | 0.60 |
| Training-free: BAGEL + GPT (Inspector) | - | 50 | 0.80 |
| Training-based: PARM (RL + TTS) | 688K | 1000 | 0.77 |
| **Ours (SFT)** | **62K** | **131** | **0.83** |

*Our method achieves superior performance with an **8x reduction in inference cost** and an **11x reduction in training data** compared to PARM.*

### Ablation Studies

**Table 5: Effect of diverse editing instructions.**
| **Case** | **Color** | **Position** | **Color Attr.** |
| :--- | :--- | :--- | :--- |
| w/o augmentation (additive only) | 0.81 | 0.58 | 0.50 |
| w/ augmentation (diverse edits) | **0.82** | **0.67** | **0.62** |
| w/ aug. + Self-critique | **0.87** | **0.72** | **0.69** |

*Diverse step instructions (refine, remove, swap) unlock more flexible reasoning, especially for relational tasks.*

**Table 6: Effect of critique construction strategy.**
| **Case** | **Color** | **Position** | **Color Attr.** |
| :--- | :--- | :--- | :--- |
| Baseline (w/ aug.) | 0.82 | 0.67 | 0.62 |
| + Scene graph corrections | 0.83 | 0.70 | 0.67 |
| **+ Self-sampling critiques** | **0.87** | **0.72** | **0.69** |

*Supervising refinement via the model's own error trajectories (self-sampling) is more effective than symbolic corrections.*

**Table 7: Complementary roles of intermediate constraints.**
| **Case** | **Counting** | **Position** | **Color Attr.** |
| :--- | :--- | :--- | :--- |
| Baseline | 0.61 | 0.66 | 0.62 |
| + Instruction-intermediate conflict (w/ ins.) | 0.62 | **0.71** | 0.65 |
| + Image-Instruction alignment (w/ img-ins.) | **0.73** | 0.69 | 0.65 |
| **w/ ins. + img-ins. (Ours)** | **0.75** | **0.72** | **0.69** |

*Semantic-level (w/ ins.) and visual-level (w/ img-ins.) constraints address distinct failure modes and are complementary.*

### Qualitative Evaluation
Figure 4 visualizes the interleaved reasoning trajectory, showing how the model detects and corrects two error types: (1) conflicts between step-level instruction and overall prompt, and (2) mismatches between the generated draft and the instruction. Figure 5 shows final images with high visual fidelity and fine-grained details.

## Theoretical and Practical Implications
*   **Theoretical:** Proves that unified multimodal models can internalize **visually-grounded reasoning**, transforming generation from a single-step commitment into a controllable, self-correcting dialogue. It demonstrates the importance of supervising **concrete visual semantics** in intermediate states rather than abstract latent noise.
*   **Practical:** Enables more **reliable, interpretable, and controllable** image synthesis, particularly for complex compositional prompts requiring precise spatial and relational reasoning. The method is **highly efficient**, achieving state-of-the-art results with a small model (7B parameters), less data, and faster inference than alternatives, making it more scalable.

## Conclusion
The paper introduces a process-driven interleaved reasoning paradigm that successfully teaches a unified multimodal model to construct images through a co-evolving loop of Plan, Sketch, Inspect, and Refine. Key breakthroughs include scene-graph subsampling, self-sampled critiques, and end-to-end training, which together enable significant performance gains on compositional and knowledge-based benchmarks with high efficiency. This work unlocks a path towards more controllable, truthful, and interpretable image synthesis. Future directions include extending this reasoning paradigm to video and 3D generation and enabling real-time human-in-the-loop control.

---

_Markdown view of https://picx.dev/p/LiWyrW, served by PicX — AI-generated visual whiteboard summaries of research papers._