Summary of "Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning"

Summary (Overview)

• Paradigm Shift: Introduces process-driven image generation, a multi-step paradigm that decomposes image synthesis into an interleaved trajectory of textual reasoning and visual actions, moving beyond single-pass, black-box generation.
• Core Process: Employs a recurring four-stage cycle—Plan, Sketch, Inspect, Refine—where textual reasoning explicitly conditions visual evolution, and generated visual intermediates ground the next round of textual reasoning.
• Key Innovations: Solves the challenge of supervising ambiguous intermediate states via scene-graph subsampling for logical trajectories, self-sampled critique data for teaching error detection, and end-to-end training of a unified multimodal model to autoregressively generate interleaved tokens.
• Performance Gains: Lifts the base BAGEL-7B model from 79% to 83% (+4%) on the GenEval benchmark for compositional alignment and from 70% to 76% (+6%) on the WISE benchmark for world knowledge reasoning.
• Efficiency: Achieves superior performance with significantly lower training data (62K vs. 688K samples) and inference cost (131 vs. 1000 sampling steps) compared to prior process-level approaches like PARM.

Introduction and Theoretical Foundation

Current text-to-image models often produce plausible but incorrect images due to the challenge of resolving complex spatial layouts, object relations, and fine-grained attributes in a single forward pass. While textual chain-of-thought (CoT) reasoning has been explored, it remains "visually blind," unable to perceive or correct spatial misalignments during generation. Existing multimodal CoT or tool-augmented approaches often decouple reasoning from generation or apply it as a post-hoc repair.

This paper challenges the outcome-driven paradigm by proposing process-driven image generation, which reformulates generation as a co-evolving trajectory of textual plans and visual states. Inspired by how humans paint incrementally, the method aims to make the generation process explicit, interpretable, and directly supervisable. The core hypothesis is that a unified multimodal model can be trained to perform interleaved reasoning, where textual and visual modalities mutually inform and constrain each other throughout a multi-step construction process, enabling error correction as it emerges.

Methodology

Framework

The model performs generation as a sequential, interleaved textual-visual reasoning process. Given a unified multimodal model $P_\theta$ and an input text prompt $T$, it generates a trajectory of alternating textual reasoning steps $s^{(i)}$ and intermediate visual states $v^{(i)}$, ultimately converging to the final image $I$:

$$\{ s^{(1)}, v^{(1)}, s^{(2)}, v^{(2)}, ..., v^{(k)}, s^{(k)} \}, I \sim P_\theta(\cdot|T) \tag{1}$$

This is realized through a recurring four-stage cycle:

1. Plan: The model generates an incremental instruction <ins> and a global scene description <des>.
2. Sketch: Conditioned on the plan, the model synthesizes an updated draft image.
3. Inspect: The model detects conflicts between (a) the textual plan/description and the original prompt, and (b) the sketch and the planned instruction.
4. Refine: If discrepancies are found, the model emits a refinement instruction <refine> and generates a corrected visual update.

The overall pipeline is summarized as:

$$T \rightarrow s^{(1)}_{plan} \rightarrow v^{(1)}_{sketch} \rightarrow s^{(1)}_{inspect} \rightarrow v^{(1)}_{refine} \rightarrow ... \rightarrow I \tag{2}$$

Intermediate Reasoning Collection

A multi-stage pipeline constructs high-quality, process-oriented reasoning traces for supervised fine-tuning.

• Multi-Turn Generation Subset: Creates the base for step-by-step generation. Uses scene-graph subsampling to derive a sequence of incremental, logically ordered prompts that expand a scene without contradictions. Augmented with GPT rewriting to enrich editing actions (modify, swap, remove).
• Instruction-Intermediate Conflict Reasoning Subset: Improves textual-side reasoning. Uses self-sampling from a fine-tuned model, with GPT as a judge to evaluate consistency with the original prompt and generate critiques for conflicts.
• Image–Instruction Alignment Reasoning Subset: Improves visual-side reasoning. Extends the Gen-Ref dataset into positive (consistent) and negative (misaligned) samples, with GPT providing explanations and corrective instructions.

Table 1: Statistics of intermediate reasoning dataset.

Model & Training

The backbone is a unified multimodal model (BAGEL-7B). It is trained to autoregressively generate interleaved sequences of text and image tokens.

• Text Loss: Cross-Entropy (CE) loss applied to textual segments $s^{(i)}$ and special vision boundary tokens. $$\mathcal{L}^{\text{text}}_{\text{CE}} = - \sum_{t \in [1,i]} \log P_\theta\left(s_t | y_{<t}, T \right) \tag{3}$$
• Image Loss: Mean Squared Error (MSE) loss based on the Rectified Flow paradigm. $$\mathbf{z}^{(i)}_t = t \cdot \mathbf{z}^{(i)}_0 + (1-t) \cdot \mathbf{z}^{(i)}_1, \quad t \in [0, 1] \tag{4}$$ $$\mathcal{L}^{\text{image}}_{\text{MSE}} = \mathbb{E}\left[ \left( P_\theta\left(\mathbf{z}^{(i)}_t | y_{<t}, T \right) - (\mathbf{z}^{(i)}_0 - \mathbf{z}^{(i)}_1) \right)^2 \right] \tag{5}$$
• Total Loss: A weighted combination. $$\mathcal{L}_{\text{total}} = \lambda_{\text{CE}} \cdot \mathcal{L}^{\text{text}}_{\text{CE}} + \mathcal{L}^{\text{image}}_{\text{MSE}} \tag{6}$$

Empirical Validation / Results

Quantitative Evaluation

Table 2: Evaluation on GenEval benchmark (compositional alignment).

Our method boosts BAGEL-7B by 5% absolute gain, achieving comparable performance to the 12B FLUX.1-dev model.

Table 3: Evaluation on WISE benchmark (world knowledge reasoning).

Our method boosts BAGEL-7B by 8.5% absolute gain, with particularly large gains on challenging domains like Time and Chemistry.

Analysis of Process-driven Reasoning

Table 4: Comparison with process-driven baselines on GenEval.

Our method achieves superior performance with an 8x reduction in inference cost and an 11x reduction in training data compared to PARM.

Ablation Studies

Table 5: Effect of diverse editing instructions.

Diverse step instructions (refine, remove, swap) unlock more flexible reasoning, especially for relational tasks.

Table 6: Effect of critique construction strategy.

Supervising refinement via the model's own error trajectories (self-sampling) is more effective than symbolic corrections.

Table 7: Complementary roles of intermediate constraints.

Semantic-level (w/ ins.) and visual-level (w/ img-ins.) constraints address distinct failure modes and are complementary.

Qualitative Evaluation

Figure 4 visualizes the interleaved reasoning trajectory, showing how the model detects and corrects two error types: (1) conflicts between step-level instruction and overall prompt, and (2) mismatches between the generated draft and the instruction. Figure 5 shows final images with high visual fidelity and fine-grained details.

Theoretical and Practical Implications

• Theoretical: Proves that unified multimodal models can internalize visually-grounded reasoning, transforming generation from a single-step commitment into a controllable, self-correcting dialogue. It demonstrates the importance of supervising concrete visual semantics in intermediate states rather than abstract latent noise.
• Practical: Enables more reliable, interpretable, and controllable image synthesis, particularly for complex compositional prompts requiring precise spatial and relational reasoning. The method is highly efficient, achieving state-of-the-art results with a small model (7B parameters), less data, and faster inference than alternatives, making it more scalable.

Conclusion

The paper introduces a process-driven interleaved reasoning paradigm that successfully teaches a unified multimodal model to construct images through a co-evolving loop of Plan, Sketch, Inspect, and Refine. Key breakthroughs include scene-graph subsampling, self-sampled critiques, and end-to-end training, which together enable significant performance gains on compositional and knowledge-based benchmarks with high efficiency. This work unlocks a path towards more controllable, truthful, and interpretable image synthesis. Future directions include extending this reasoning paradigm to video and 3D generation and enabling real-time human-in-the-loop control.