# DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes

> DecoupleMix decouples VLM data mixture optimization into inter-class ratio search and convex intra-class allocation, achieving competitive performance at 1/80th the multimodal training tokens of stronger baselines.

- **Source:** [arXiv](https://arxiv.org/abs/2607.24516)
- **Published:** 2026-09-05
- **Permalink:** https://picx.dev/p/8eZ0oQ
- **Whiteboard:** https://picx.dev/p/8eZ0oQ/image

## Summary

# DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes

## Summary (Overview)

- **Decoupled mixture optimization framework**: The paper introduces DecoupleMix, a systematic framework that decomposes VLM data mixture construction into two orthogonal sub-problems: inter-class ratio allocation (across capability categories) and intra-class composition(within a category), making data recipe optimization tractable and reproducible.
- **Automated dataset-level assessment**: Uses an LLM-as-a-Judge paradigm with Seed-1.6 to score datasets on quality(4 dimensions)and difficulty(6 dimensions), validated against human consensus(Spearman $r = 0 . 8 2$ for Quality, $r = 0 . 7 5$ for Difficulty)．
- **Convex optimization for intra-class allocation**: Formulates intra-class sampling as a constrained convex optimization problem with quality, difficulty, and entropy-based diversity objectives, solved via ECOS.

- **Attributable data admission**: Establishesa controlled protocol where admitting a new dataset under fixed inter-class ratios and budget yields an effect attributable to that dataset alone, reducing global confounding compared to naive "re-stack and retrain" approaches．
- **Scalability and transferability**: Optimal ratios discovered at small proxy scales(2.5B tokens)transfer seamlessly to larger data scales(5B, 10B)and model scales(32B parameters)without retuning, achieving competitive performance against strong open-source baselines with 80B additional multimodal tokens．

## Introduction and Theoretical Foundation

### Background and Motivation

Vision Language Models(VLMs)have become central to general-purpose multimodal systems,with proprietary models like GPT-5.6,Seed-2.0,and Gemini-3.5-Flash demonstrating remarkable capabilities．However,a considerable performance gap persists between frontier proprietary models and open-source counterparts．Beyond architecture and scale,growing evidence highlights the **Data Mixture Recipe**—the selection,composition,and allocation of training data—as a critical determinant of model capability and data efficiency．

### Key Limitations of Current Practice

The paper identifies two critical limitations in existing open-source pipelines:

1. **Over-reliance on Quality Filtering**: Prevailing methods focus on cleaning "noisy"data,but lack systematic assessment of Data Difficulty and Diversity．
2. **Unsystematic Mixture Strategies**: Cross-domain ratios are typically set by intuition,and intra-class composition relies on brute-force stacking of all filtered data,ignoring redundancy．

### Theoretical Foundation

The central insight is that the monolithic question of"what to mix"can be **decoupled into two hierarchical sub-problems**:

- **Inter-class allocation**: How to split the token budget across capability categories(e.g.,Grounding vs．Caption)．
- **Intra-class composition**: How to mix candidate sources within a single category(e.g.,multiple OCR datasets)．

This decoupling mirrors industrial data organization and sidesteps the intractability of high-dimensional search．Rather than sample-level filtering,the framework scores each candidate dataset as a whole unit,keeping optimization tractable across hundreds of sources．

## Methodology

### 2.1 Automated Dataset-Level Assessment

The framework quantifies three properties for each dataset:

- **Capability**: Decomposed into an extensible hierarchical taxonomy,with automated assignment via natural-language criteria．
- **Quality score** $q_i$: Weighted average of four dimensions(penalizing hallucinations)．
- **Difficulty score** $d_i$: Aggregates six dimensions(prioritizing cross-modal synthesis and prior knowledge)．

**Implementation**: Uses an LLM-as-a-Judge paradigm with Seed-1.6 as evaluator,sampling 150 instances per dataset．Diversity is enforced structurally via an entropy term during mixture optimization rather than scored per dataset．

### 2.2 Decoupled Mixture Optimization

#### Inter-class Strategy

Global ratios between broad categories are treated as hyperparameters and searched via a coordinate-style single-variable procedure:

- Start from a warm-start configuration $\mathbf{r}^{(0)}$(the average of reference recipes)．
- Each round sweeps one category's ratio $r_c$ over a candidate set while holding others fixed(renormalizing the remainder)．
- Keep the value maximizing validation performance; iterate until stabilization．

This optimizes ratios in linear rather than exponential time．

#### Intra-class Convex Optimization

For granular data selection within a category,the framework solves a **Constrained Convex Optimization problem** independently per category $c$ under its allocated budget $T_c = r_c T$:

$$ \begin{array}{ll} \max_{\mathbf{w} \in \Delta^{N-1}}} & \sum_{i=1}^{N} w_i \big(\alpha q_i + \beta d_i \big) + \gamma \mathcal{H}(\mathbf{w}) \\ \text{s.t.} & \ell_i \leq w_i \leq u_i, \qquad i=1,\ldots,N, \end{array}\tag{1} $$

where

$$ \Delta^{N-1} = \left\{\mathbf{w} \in \mathbb{R}_{+}^{N} \middle| \sum_{i=1}^{N} w_i = 1 \right\}, \quad \mathcal{H}(\mathbf{w}) = - \sum_{i=1}^{N} w_i \log w_i.\tag{2} $$

Here,$q_i$ and $d_i$ denote quality and difficulty scores of dataset $i$;$\alpha,\beta,\gamma \ge 0$ control relative contributions;$\ell_i = L_i / T_c$ and $u_i = M_i / T_c$ are normalized lowerand upper allocation bounds,where $L_i$ denotes token-level bounds and $M_i$ denotes total available tokens in dataset $i$．The resulting token allocation is:

$$ t_i = T_c w_i, \qquad \sum_{i=1}^{N} t_i = T_c.\tag{3} $$

The entropy regularizer $\mathcal{H}(\mathbf{w})$ discourages excessive concentration on high-scoring datasets and promotes broader coverage．Since $\mathcal{H}(\mathbf{w})$ is concave and the feasible region is convex,Eq．(1)defines a convex optimization problem,solved using ECOS．

### 2.3 Data Acceptance via Attributable Validation

The protocol creates a controlled admission comparison making dataset admission mathematically identifiable．Let performance be $P = f(\mathbf{r},\mathbf{x},T)$,where $\mathbf{r}$ is inter-class ratio,$T$ the budget,and $\mathbf{x}$ intra-class allocation．To first order:

$$ \mathrm{d}P = \nabla_{\mathbf{r}} f \cdot \mathrm{d}\mathbf{r} + \sum_{c} \nabla_{\mathbf{x}_c} f \cdot \mathrm{d}\mathbf{x}_c + \frac{\partial f}{\partial T} \mathrm{d}T.\tag{4} $$

The protocol fixes $\mathbf{r},T$,and optimizer $g$,while admitting the candidate into exactly one class $c$．The induced shift $\mathrm{d}\mathbf{x}_c$ is part of the response to a single admission intervention:

$$ \Delta P_{d^{\star}} = f\big(\mathbf{r}, g(\mathcal{D} \cup \{d^{\star}\}), T\big) - f\big(\mathbf{r}, g(\mathcal{D}), T\big)\tag{5} $$

This captures the total effect of admittingthe dataset—its direct contribution plus budget-preserving re-balancing within its class．

### Training Stages

1. **Alignment**(8B tokens):High-quality image-caption pairs align visual features with the LLM;only the projector is trainable．
2. **Mixture Validation**: Full-parameter fine-tuning on mixed datasets to assess comprehensive capabilities．
3. **Annealing**: Short,high-quality refinement stage with learning rate decaying to zero．

Models are assembled from pretrained unimodal-stage components(Qwen3-VL-ViT visual encoder + text-only Qwen3-4B-Instruct language backbone + randomly initialized projector)rather than fine-tuning an existing VLM．

## Empirical Validation / Results

### 4.1 Main Results

Models trained with 80B additional multimodal continued-pretraining tokens are compared against Qwen3-VL-Instruct baselines(Table 1)：

**Table  ̄1: Comprehensive Performance Comparison**

| Models | Params． | MMBench | MME | MMStar | R．WorldQA | MMMU | BLINK | HalBench | AI2D | OCRBen． | InfoVQA | CharXiv | LogicVis | VisuLog | PuzzleVQA | VMME | MVBench | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-VL-2B | 2B | 77.4 | 72.5 | 56.1 | 64.1 | 49.6 | 55.3 | 45.1 | 75.8 | 82.2 | 58.2 | 51.6 | 35.6 | 24.4 | 13.4 | 47.8 | 50.2 | 53.7 |
| **Our-1B** | **1B** | 72.2 | 72.0 | 54.2 | 62.1 | 41.0 | 52.9 | 44.6 | 71.2 | 80.4 | 52..9 | 48..6 | 31..5 | 24..8 | 30..2 | 45..1 | 53..6 |**52..3** |
| Qwen3-VL-4B | 4B | 83..7 | 82..0 | 66..9 | 70..1 | 60..0 | 64..4 |

---

_Markdown view of https://picx.dev/p/8eZ0oQ, served by PicX — AI-generated visual whiteboard summaries of research papers._