Summary of "Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding?"

Summary (Overview)

• Novel self-recovery paradigm: Proposes Robust-U1, the first framework that equips Multimodal Large Language Models (MLLMs) with explicit visual self-recovery capability, directly reconstructing clean images from corrupted inputs rather than relying on implicit feature alignment or text-only reasoning.
• Three-stage training pipeline: (i) Supervised fine-tuning (SFT) on ImageNet-C for foundational reconstruction, (ii) Reinforcement learning (Flow-GRPO) with dual rewards — pixel-level SSIM and semantic-level CLIP similarity — to align structural and semantic fidelity, (iii) Multimodal reasoning that jointly considers both corrupted and recovered images for robust understanding.
• State-of-the-art performance: Achieves significant improvements on R-Bench (overall 0.7398 vs. 0.5770 for base BAGEL) and maintains minimal performance drops under adversarial corruptions on MMMB, MMStar, and RealWorldQA.
• Self-recovery enhances reasoning: Quantitative analysis confirms that high-quality visual recovery directly improves downstream reasoning performance, validating self-recovery as a critical mechanism for robust visual understanding.
• Code released: Source code available at github.com/jqtangust/Robust-U1.

Introduction and Theoretical Foundation

Background and Motivation

Multimodal Large Language Models (MLLMs) have achieved remarkable success in visual understanding through large-scale pretraining. However, their real-world deployment is critically hindered by vulnerability to visual corruptions — such as system noise, compression artifacts, and adverse weather — that severely disrupt visual features and cause dramatic performance degradation.

Limitations of Existing Approaches

Existing robustness enhancement methods fall into two categories:

1. Black-box feature alignment (e.g., TeCoA, Robust CLIP, Robust LLaVA): Align features of corrupted and clean images within the visual encoder via adversarial training. Limitation: lacks interpretability and fails to explicitly model the corruption process.

2. White-box text-based reasoning (e.g., Robust-R1): Uses explicit textual chains to describe corruption types and semantic impacts. Limitation: cannot represent pixel-level details; textual descriptions cannot restore lost visual information.

Research Question

Can MLLMs recover corrupted visual content by themselves?

The paper proposes that achieving self-recovery — where the model actively restores lost pixel-level information rather than merely offering text-based compensation — would establish a more intrinsic and complete form of robustness.

Theoretical Formulation

Consider a standard multimodal pipeline with clean image $I_o \in \mathbb{R}^{H \times W \times 3}$ and query $Q$:

$$A_o = \mathcal{F}_{\text{MLLM}}(I_o, Q; \Theta)$$

where $\Theta$ denotes model parameters. Under real-world corruption $I_c = \mathcal{D}(I_o)$, where $\mathcal{D}$ is the corruption function, performance degrades significantly.

The proposed robust formulation explicitly incorporates a visual self-recovery process:

$$A = \mathcal{F}_{\text{MLLM}}^{\text{(Robust)}}(I_c, Q; \Theta) = \mathcal{F}_{\text{MLLM}}\left(\underbrace{\mathcal{D}^{-1}(I_c)}_{I_r}, I_c, Q; \Theta\right)$$

where $\mathcal{D}^{-1}: I_c \mapsto I_r$ represents the self-recovery module approximating the inverse corruption process, and the model synthesizes information from both $I_c$ and $I_r$ for robust reasoning.

Methodology

Overview of Three-Stage Framework

The framework is built on BAGEL, a pre-trained unified MLLM supporting both multimodal understanding and generation.

Stage I: Supervised Fine-Tuning for Visual Self-Recovery (SFT)

• Goal: Transform the model's general generative capability into a dedicated visual self-recovery module.
• The recovery process is conditioned on a specific recovery prompt $P_{\text{rec}}$.
• Uses rectified flow formulation (Liu et al., 2023) in latent space.
• Image is encoded into latent representation $Z_c$; model denoises a noisy version of clean latent $Z_o$ conditioned on $Z_c$ and $P_{\text{rec}}$.
• Objective function:

$$L_{\text{SFT}} = \mathbb{E}_{t \sim \mathcal{U}(0,1), \epsilon \sim \mathcal{N}(0, I)} \left[ \| \epsilon - \epsilon_\Theta(Z_c, Z_o(t), t, P_{\text{rec}}) \|^2 \right]$$

where $Z_o(t) = (1-t)Z_o + t\epsilon$ is the noisy latent at timestep $t$, and $\epsilon_\Theta$ is the noise prediction network.

Stage II: Aligning Higher Visual Quality through Reinforcement Learning

Uses Flow-GRPO (Liu et al., 2025b) to optimize the self-recovery module with a dual-reward objective:

Pixel-Level Structural Reward — SSIM (Structural Similarity Index Measure):

$$R_{\text{pix}}(I_r, I_o) = \text{SSIM}(I_r, I_o) = \frac{1}{N} \sum_{i=1}^N \left[ l(p_r^i, p_o^i) \cdot c(p_r^i, p_o^i) \cdot s(p_r^i, p_o^i) \right]$$

where the three components for luminance, contrast, and structure are:

$$l(p_r, p_o) = \frac{2\mu_r \mu_o + C_1}{\mu_r^2 + \mu_o^2 + C_1}, \quad c(p_r, p_o) = \frac{2\sigma_r \sigma_o + C_2}{\sigma_r^2 + \sigma_o^2 + C_2}, \quad s(p_r, p_o) = \frac{\sigma_{ro} + C_3}{\sigma_r \sigma_o + C_3}$$

with $\mu_r, \mu_o$ patch means, $\sigma_r, \sigma_o$ patch standard deviations, $\sigma_{ro}$ covariance, and $C_1, C_2, C_3$ small constants for numerical stability.

Semantic Consistency Reward — uses frozen TinyCLIP to compute cosine similarity:

$$\text{Sim}(M_{\text{CLIP}}(I_r), M_{\text{CLIP}}(I_o)) = \frac{M_{\text{CLIP}}(I_r) \cdot M_{\text{CLIP}}(I_o)}{\|M_{\text{CLIP}}(I_r)\| \|M_{\text{CLIP}}(I_o)\|}$$ $$R_{\text{sem}}(I_r, I_o) = \exp\left(-\alpha \cdot (1 - \text{Sim}(M_{\text{CLIP}}(I_r), M_{\text{CLIP}}(I_o)))\right)$$

where $\alpha > 0$ controls sensitivity to semantic deviations. The reward is maximized (1) when similarity is 1, decaying exponentially as similarity decreases.

Optimization: For each $I_c$, sample $G$ trajectories (via stochastic ODE-to-SDE conversion) to get $\{I_r^i\}_{i=1}^G$. Advantages computed via group normalization of composite rewards; policy updated with KL divergence penalty.

Stage III: Multimodal Reasoning for Robust Understanding

• Structure input as interleaved sequence of corrupted image $I_c$ and recovered image $I_r$, followed by query $Q$.
• Train model to generate answer $A$ (with reasoning chain) conditioned on this input.
• Training objective (next-token prediction):

$$L_{\text{MLLM}} = -\mathbb{E}_{(I_c, I_r, Q, A^*)} \sum_{t=1}^L \log P_\Theta(a_t^* | a_{<t}^*, I_c, I_r, Q)$$

Empirical Validation / Results

Experimental Setup

• Base model: BAGEL (unified MLLM)
• Training data: SFT on ImageNet-C; RL and multimodal reasoning on Robust-R1 training data
• Benchmarks:
  • Real-world corruption: R-Bench (MCQ, VQA, CAP tasks across low/mid/high degradation)
  • Adversarial corruption: MMMB, MMStar, RealWorldQA (with synthetic degradations at 25%, 50%, 100% intensity)
• Baselines: General MLLMs (Qwen2.5-VL-3B, Gemma3-4B, InternVL-4B, BAGEL) and Robust MLLMs (TeCoA, Robust CLIP, Robust LLaVA, Robust-R1)

Main Results – Real-World Corruptions

Table 1. Quantitative evaluation on R-Bench. Best/Red, second best/Blue.

Robust-U1 outperforms all baselines across all tasks and intensities. The advantage becomes more pronounced as corruption severity increases.

Main Results – Adversarial Corruptions

Table 3. Quantitative evaluation on anti-degradation for MMMB, MMStar, RealWorldQA. Best/Red, second best/Blue.

Robust-U1 achieves minimal performance drops: only 1.57 points on MMMB from clean to 100% corruption (vs. 3.44 for BAGEL and 6.06 for Robust-R1).

Quality of Visual Recovery

Table 5. Quantitative evaluation of visual recovery quality. Best/Red, second best/Blue.

Each stage contributes to improved recovery. The full model with both rewards achieves the best overall balance across all metrics.

Ablation Study

Table 4. Ablation on R-Bench. Best/Red, second best/Blue.