Summary (Overview)

• RationalRewards is a novel reward model for visual generation that produces structured, multi-dimensional natural language critiques (rationales) before assigning scores, moving beyond traditional scalar outputs.
• The Preference-Anchored Rationalization (PARROT) framework enables training such a model without costly rationale annotations by treating rationales as latent variables and deriving them from widely available pairwise preference data via a three-phase pipeline.
• The model enables optimization in two complementary spaces: as a fine-grained, interpretable reward for Reinforcement Learning (RL) in parameter space, and as the core of a test-time Generate–Critique–Refine loop for prompt-space optimization without parameter updates.
• Empirical results show the 8B-parameter RationalRewards achieves state-of-the-art preference prediction among open-source models, is competitive with large commercial models (Gemini-2.5-Pro), and uses 10–20× less training data. Most strikingly, its test-time prompt-tuning loop matches or exceeds the gains from expensive RL fine-tuning on several benchmarks.

Introduction and Theoretical Foundation

The advancement of visual generation models is increasingly constrained by the quality of reward models used to evaluate their outputs. Current standard reward models act as scalar black boxes, compressing rich, multi-dimensional human judgments (e.g., perceptual quality, instruction faithfulness) into a single unexplained score. This discards the structured reasoning underlying human preference and can lead to issues like reward hacking, where generators exploit shortcut correlations in the reward signal rather than learning principled evaluation criteria.

This paper asks: can reward models be made to reason? The authors introduce RationalRewards, a reasoning-based reward model that generates explicit, structured critiques before deriving scores. This transforms the model from a passive evaluator into an active optimization interface. The key theoretical shift is from modeling $P(y|x)$ directly to introducing a latent natural language rationale $z$ that explains the preference $y$. The goal is to learn a model $P_θ(z, y|x)$. To learn this from preference data alone (without rationale annotations), the authors propose the Preference-Anchored Rationalization (PARROT) framework, which formulates an Evidence Lower Bound (ELBO) to recover high-quality rationales.

Methodology

1. Variational Framework: Preference-Anchored Rationalization (PARROT)

The core problem is learning a reward model $P_θ(z, y|x)$ from pairwise preference data $\{(x, y)\}$, where $x = (I_A, I_B, c)$ is a comparison tuple and $y ∈ \{A≻B, B≻A\}$, without ground-truth rationales $z$.

The authors derive an Evidence Lower Bound (ELBO):

$$\mathcal{L}_{ELBO} = \mathbb{E}_{z∼q_φ} [\log P_θ(y|x, z)] - D_{KL}(q_φ(z|x, y) \| P_θ(z|x))$$

Term 1 (Prediction): Rationale $z$ should be predictive of preference $y$. Term与新2 (Regularization): The student model's prior $P_θ(z|x)$ should match the inferred posterior $q_φ(z|x, y)$.

This decomposition maps to a practical three-phase pipeline:

• Phase 1: Rationale Generation (Constructing $q_φ(z|x, y)$). A teacher VLM (Qwen3-VL-32B-Instruct) generates candidate rationales, anchored to the known preference label $y$, to focus on justifications consistent with the human judgment.
• Phase 2: Predictive Consistency Filtering (Maximizing Term 1). The teacher is re-queried with the rationale $z$ (without the label $y$) to verify it can recover the preference. A consistency check $C(x,y,z)$ retains only rationales that are genuinely predictive: $$C(x, y, z) = \mathbb{I}\left[\arg \max_{y'} P_{\text{Teacher}}(y'|x, z) = y\right]$$
• Phase 3: Foresight Learning (Minimizing Term 2). A student model $P_θ(z|x)$ is trained via Supervised Fine-Tuning (SFT) on the filtered rationale samples $(x, z)$, learning to generate rationales without seeing the answer.

2. Pointwise Projection for Deployment

Downstream tasks (RL, test-time critique) require pointwise feedback on individual images. The authors use a Pointwise Projection Strategy: the teacher VLM assesses each image in isolation, using the validated pairwise rationale as a reference hint to guide attention, producing absolute scores (1-4 scale) across four dimensions: Text Faithfulness, Image Faithfulness, Physical & Visual Quality, and Text Rendering. The student is trained jointly on both pairwise and projected pointwise data.

3. Dual-Space Optimization

The rationalized reward model enables optimization in two spaces:

• Parameter Space (RL Fine-Tuning): Multi-dimensional scores provide semantically decomposed reward signals for RL (using the DiffusionNFT algorithm), replacing a single opaque scalar.
• Prompt Space (Test-Time Refinement): The Generate–Critique–Refine loop uses RationalRewards' natural-language critique to identify deficiencies and generate a targeted prompt revision for re-generation, optimizing $t^* = \arg \max_t R(G(t))$ guided by language.

Empirical Validation / Results

1. Preference Modeling Accuracy

RationalRewards (8B) surpasses all open-source scalar reward models across multiple benchmarks and is competitive with large commercial models.

Table 1: Comparison of reward models as evaluators (pairwise accuracy %).

An ablation confirms the contribution of PARROT: direct SFT distillation from the 32B teacher to the 8B student performs significantly worse than RationalRewards.

2. Optimization in Dual Spaces

Parameter Space (RL): RL fine-tuning guided by RationalRewards consistently outperforms baselines using scalar rewards (EditReward, MultiReward) or a generic VLM judge (Qwen3-VL-32B).

Table 2: Text-to-image RL on UniGenBench++ (Overall Score).

Table 3 (Right): Image editing benchmarks (Overall Score).

Prompt Space (Test-Time): The Generate–Critique–Refine loop (PT in tables) adds only ~0.4 seconds of overhead. Remarkably, its improvements match or exceed those from computationally expensive RL fine-tuning on several benchmarks (e.g., raising Flux.1 Kontext from 3.52 to 4.01 on ImgEdit-Bench).

3. Additional Findings

• Resistance to Reward Hacking: RL with scalar rewards shows reward increases while generation quality degrades. RationalRewards maintains a monotonic correspondence between reward and quality, as producing coherent rationales structurally grounds the evaluation.
• Scoring Stability: Unlike generic VLMs used as judges, RationalRewards produces low-variance, preference-aligned scores, leading to more stable RL optimization.
• Data Efficiency: RationalRewards achieves high performance using only 80K raw preference pairs (57.6K after filtering), which is 10-20x less data than comparable baselines.

Theoretical and Practical Implications

• Theoretical: The PARROT framework provides a principled, variational method for distilling reasoning capabilities from preference data, connecting latent variable modeling to a scalable pipeline. It demonstrates that structured rationalization acts as a powerful inductive bias, enabling small models to achieve high accuracy.
• Practical for Model Development: RationalRewards serves as a superior, interpretable reward signal for RL that mitigates reward hacking. It also enables test-time quality boosting via prompt refinement, a cost-effective alternative to fine-tuning.
• Practical for End-Users: The Generate–Critique–Refine loop can help users elicit latent capabilities from existing generators by automatically refining suboptimal prompts, democratizing access to high-quality outputs.
• Broader Paradigm Shift: The work suggests a shift from scalar regression to rationalization in reward modeling, emphasizing transparency and reasoning. It also highlights test-time compute scaling as a viable axis for improvement orthogonal to parameter-space training.

Conclusion

The paper presents RationalRewards, a reasoning-based reward model trained via the PARROT framework. The key conclusions are:

1. Structured rationalization enables an 8B model to achieve preference-prediction accuracy competitive with much larger commercial models while using significantly less data.
2. The resulting multi-dimensional rationales provide superior, interpretable rewards for RL that resist reward hacking.
3. Most notably, the test-time Generate–Critique–Refine loop matches or exceeds RL fine-tuning, providing strong empirical support for the latent capability hypothesis—that current generators possess under-elicited potential that structured critiques can unlock.

The authors release models and code to facilitate further research. Future directions include extending the approach to other modalities (video, 3D), reducing teacher model dependence, and conducting comprehensive bias audits.