# Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding? > Robust-U1 enables MLLMs to explicitly self-recover corrupted images, achieving state-of-the-art robust understanding across real-world and adversarial corruptions. - **Source:** [arXiv](https://arxiv.org/abs/2606.08063) - **Published:** 2026-06-13 - **Permalink:** https://picx.dev/p/H41geU - **Whiteboard:** https://picx.dev/p/H41geU/image ## Summary # Summary of "Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding?" ## Summary (Overview) - **Novel self-recovery paradigm**: Proposes **Robust-U1**, the first framework that equips Multimodal Large Language Models (MLLMs) with explicit visual self-recovery capability, directly reconstructing clean images from corrupted inputs rather than relying on implicit feature alignment or text-only reasoning. - **Three-stage training pipeline**: (i) Supervised fine-tuning (SFT) on ImageNet-C for foundational reconstruction, (ii) Reinforcement learning (Flow-GRPO) with dual rewards — pixel-level SSIM and semantic-level CLIP similarity — to align structural and semantic fidelity, (iii) Multimodal reasoning that jointly considers both corrupted and recovered images for robust understanding. - **State-of-the-art performance**: Achieves significant improvements on R-Bench (overall 0.7398 vs. 0.5770 for base BAGEL) and maintains minimal performance drops under adversarial corruptions on MMMB, MMStar, and RealWorldQA. - **Self-recovery enhances reasoning**: Quantitative analysis confirms that high-quality visual recovery directly improves downstream reasoning performance, validating self-recovery as a critical mechanism for robust visual understanding. - **Code released**: Source code available at [github.com/jqtangust/Robust-U1](https://github.com/jqtangust/Robust-U1). ## Introduction and Theoretical Foundation ### Background and Motivation Multimodal Large Language Models (MLLMs) have achieved remarkable success in visual understanding through large-scale pretraining. However, their real-world deployment is critically hindered by vulnerability to visual corruptions — such as system noise, compression artifacts, and adverse weather — that severely disrupt visual features and cause dramatic performance degradation. ### Limitations of Existing Approaches Existing robustness enhancement methods fall into two categories: 1. **Black-box feature alignment** (e.g., TeCoA, Robust CLIP, Robust LLaVA): Align features of corrupted and clean images within the visual encoder via adversarial training. *Limitation*: lacks interpretability and fails to explicitly model the corruption process. 2. **White-box text-based reasoning** (e.g., Robust-R1): Uses explicit textual chains to describe corruption types and semantic impacts. *Limitation*: cannot represent pixel-level details; textual descriptions cannot restore lost visual information. ### Research Question > Can MLLMs recover corrupted visual content by themselves? The paper proposes that achieving **self-recovery** — where the model actively restores lost pixel-level information rather than merely offering text-based compensation — would establish a more intrinsic and complete form of robustness. ### Theoretical Formulation Consider a standard multimodal pipeline with clean image $I_o \in \mathbb{R}^{H \times W \times 3}$ and query $Q$: $$A_o = \mathcal{F}_{\text{MLLM}}(I_o, Q; \Theta)$$ where $\Theta$ denotes model parameters. Under real-world corruption $I_c = \mathcal{D}(I_o)$, where $\mathcal{D}$ is the corruption function, performance degrades significantly. The proposed robust formulation explicitly incorporates a visual self-recovery process: $$A = \mathcal{F}_{\text{MLLM}}^{\text{(Robust)}}(I_c, Q; \Theta) = \mathcal{F}_{\text{MLLM}}\left(\underbrace{\mathcal{D}^{-1}(I_c)}_{I_r}, I_c, Q; \Theta\right)$$ where $\mathcal{D}^{-1}: I_c \mapsto I_r$ represents the self-recovery module approximating the inverse corruption process, and the model synthesizes information from both $I_c$ and $I_r$ for robust reasoning. ## Methodology ### Overview of Three-Stage Framework The framework is built on **BAGEL**, a pre-trained unified MLLM supporting both multimodal understanding and generation. #### Stage I: Supervised Fine-Tuning for Visual Self-Recovery (SFT) - Goal: Transform the model's general generative capability into a dedicated visual self-recovery module. - The recovery process is conditioned on a specific recovery prompt $P_{\text{rec}}$. - Uses **rectified flow formulation** (Liu et al., 2023) in latent space. - Image is encoded into latent representation $Z_c$; model denoises a noisy version of clean latent $Z_o$ conditioned on $Z_c$ and $P_{\text{rec}}$. - Objective function: $$L_{\text{SFT}} = \mathbb{E}_{t \sim \mathcal{U}(0,1), \epsilon \sim \mathcal{N}(0, I)} \left[ \| \epsilon - \epsilon_\Theta(Z_c, Z_o(t), t, P_{\text{rec}}) \|^2 \right]$$ where $Z_o(t) = (1-t)Z_o + t\epsilon$ is the noisy latent at timestep $t$, and $\epsilon_\Theta$ is the noise prediction network. #### Stage II: Aligning Higher Visual Quality through Reinforcement Learning Uses **Flow-GRPO** (Liu et al., 2025b) to optimize the self-recovery module with a dual-reward objective: **Pixel-Level Structural Reward** — SSIM (Structural Similarity Index Measure): $$R_{\text{pix}}(I_r, I_o) = \text{SSIM}(I_r, I_o) = \frac{1}{N} \sum_{i=1}^N \left[ l(p_r^i, p_o^i) \cdot c(p_r^i, p_o^i) \cdot s(p_r^i, p_o^i) \right]$$ where the three components for luminance, contrast, and structure are: $$l(p_r, p_o) = \frac{2\mu_r \mu_o + C_1}{\mu_r^2 + \mu_o^2 + C_1}, \quad c(p_r, p_o) = \frac{2\sigma_r \sigma_o + C_2}{\sigma_r^2 + \sigma_o^2 + C_2}, \quad s(p_r, p_o) = \frac{\sigma_{ro} + C_3}{\sigma_r \sigma_o + C_3}$$ with $\mu_r, \mu_o$ patch means, $\sigma_r, \sigma_o$ patch standard deviations, $\sigma_{ro}$ covariance, and $C_1, C_2, C_3$ small constants for numerical stability. **Semantic Consistency Reward** — uses frozen TinyCLIP to compute cosine similarity: $$\text{Sim}(M_{\text{CLIP}}(I_r), M_{\text{CLIP}}(I_o)) = \frac{M_{\text{CLIP}}(I_r) \cdot M_{\text{CLIP}}(I_o)}{\|M_{\text{CLIP}}(I_r)\| \|M_{\text{CLIP}}(I_o)\|}$$ $$R_{\text{sem}}(I_r, I_o) = \exp\left(-\alpha \cdot (1 - \text{Sim}(M_{\text{CLIP}}(I_r), M_{\text{CLIP}}(I_o)))\right)$$ where $\alpha > 0$ controls sensitivity to semantic deviations. The reward is maximized (1) when similarity is 1, decaying exponentially as similarity decreases. **Optimization**: For each $I_c$, sample $G$ trajectories (via stochastic ODE-to-SDE conversion) to get $\{I_r^i\}_{i=1}^G$. Advantages computed via group normalization of composite rewards; policy updated with KL divergence penalty. #### Stage III: Multimodal Reasoning for Robust Understanding - Structure input as interleaved sequence of corrupted image $I_c$ and recovered image $I_r$, followed by query $Q$. - Train model to generate answer $A$ (with reasoning chain) conditioned on this input. - Training objective (next-token prediction): $$L_{\text{MLLM}} = -\mathbb{E}_{(I_c, I_r, Q, A^*)} \sum_{t=1}^L \log P_\Theta(a_t^* | a_{