DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes

Summary (Overview)

Decoupled mixture optimization framework: The paper introduces DecoupleMix, a systematic framework that decomposes VLM data mixture construction into two orthogonal sub-problems: inter-class ratio allocation (across capability categories) and intra-class composition(within a category), making data recipe optimization tractable and reproducible.
Automated dataset-level assessment: Uses an LLM-as-a-Judge paradigm with Seed-1.6 to score datasets on quality(4 dimensions)and difficulty(6 dimensions), validated against human consensus(Spearman $r = 0 . 8 2$ for Quality, $r = 0 . 7 5$ for Difficulty)．
Convex optimization for intra-class allocation: Formulates intra-class sampling as a constrained convex optimization problem with quality, difficulty, and entropy-based diversity objectives, solved via ECOS.
Attributable data admission: Establishesa controlled protocol where admitting a new dataset under fixed inter-class ratios and budget yields an effect attributable to that dataset alone, reducing global confounding compared to naive "re-stack and retrain" approaches．
Scalability and transferability: Optimal ratios discovered at small proxy scales(2.5B tokens)transfer seamlessly to larger data scales(5B, 10B)and model scales(32B parameters)without retuning, achieving competitive performance against strong open-source baselines with 80B additional multimodal tokens．

Introduction and Theoretical Foundation

Background and Motivation

Vision Language Models(VLMs)have become central to general-purpose multimodal systems,with proprietary models like GPT-5.6,Seed-2.0,and Gemini-3.5-Flash demonstrating remarkable capabilities．However,a considerable performance gap persists between frontier proprietary models and open-source counterparts．Beyond architecture and scale,growing evidence highlights the Data Mixture Recipe—the selection,composition,and allocation of training data—as a critical determinant of model capability and data efficiency．

Key Limitations of Current Practice

The paper identifies two critical limitations in existing open-source pipelines:

Over-reliance on Quality Filtering: Prevailing methods focus on cleaning "noisy"data,but lack systematic assessment of Data Difficulty and Diversity．
Unsystematic Mixture Strategies: Cross-domain ratios are typically set by intuition,and intra-class composition relies on brute-force stacking of all filtered data,ignoring redundancy．

Theoretical Foundation

The central insight is that the monolithic question of"what to mix"can be decoupled into two hierarchical sub-problems:

Inter-class allocation: How to split the token budget across capability categories(e.g.,Grounding vs．Caption)．
Intra-class composition: How to mix candidate sources within a single category(e.g.,multiple OCR datasets)．

This decoupling mirrors industrial data organization and sidesteps the intractability of high-dimensional search．Rather than sample-level filtering,the framework scores each candidate dataset as a whole unit,keeping optimization tractable across hundreds of sources．

Methodology

2.1 Automated Dataset-Level Assessment

The framework quantifies three properties for each dataset:

Capability: Decomposed into an extensible hierarchical taxonomy,with automated assignment via natural-language criteria．
Quality score $q_i$ : Weighted average of four dimensions(penalizing hallucinations)．
Difficulty score $d_i$ : Aggregates six dimensions(prioritizing cross-modal synthesis and prior knowledge)．

Implementation: Uses an LLM-as-a-Judge paradigm with Seed-1.6 as evaluator,sampling 150 instances per dataset．Diversity is enforced structurally via an entropy term during mixture optimization rather than scored per dataset．

2.2 Decoupled Mixture Optimization

Inter-class Strategy

Global ratios between broad categories are treated as hyperparameters and searched via a coordinate-style single-variable procedure:

Start from a warm-start configuration $\mathbf{r}^{(0)}$ (the average of reference recipes)．
Each round sweeps one category's ratio $r_c$ over a candidate set while holding others fixed(renormalizing the remainder)．
Keep the value maximizing validation performance; iterate until stabilization．

This optimizes ratios in linear rather than exponential time．

Intra-class Convex Optimization

For granular data selection within a category,the framework solves a Constrained Convex Optimization problem independently per category $c$ under its allocated budget $T_c = r_c T$ :

\begin{array}{ll} \max_{\mathbf{w} \in \Delta^{N-1}}} & \sum_{i=1}^{N} w_i \big(\alpha q_i + \beta d_i \big) + \gamma \mathcal{H}(\mathbf{w}) \\ \text{s.t.} & \ell_i \leq w_i \leq u_i, \qquad i=1,\ldots,N, \end{array}\tag{1}

where

\Delta^{N-1} = \left\{\mathbf{w} \in \mathbb{R}_{+}^{N} \middle| \sum_{i=1}^{N} w_i = 1 \right\}, \quad \mathcal{H}(\mathbf{w}) = - \sum_{i=1}^{N} w_i \log w_i.\tag{2}

Here, $q_i$ and $d_i$ denote quality and difficulty scores of dataset $i$ ; $\alpha,\beta,\gamma \ge 0$ control relative contributions; $\ell_i = L_i / T_c$ and $u_i = M_i / T_c$ are normalized lowerand upper allocation bounds,where $L_i$ denotes token-level bounds and $M_i$ denotes total available tokens in dataset $i$ ．The resulting token allocation is:

t_i = T_c w_i, \qquad \sum_{i=1}^{N} t_i = T_c.\tag{3}

The entropy regularizer $\mathcal{H}(\mathbf{w})$ discourages excessive concentration on high-scoring datasets and promotes broader coverage．Since $\mathcal{H}(\mathbf{w})$ is concave and the feasible region is convex,Eq．(1)defines a convex optimization problem,solved using ECOS．

2.3 Data Acceptance via Attributable Validation

The protocol creates a controlled admission comparison making dataset admission mathematically identifiable．Let performance be $P = f(\mathbf{r},\mathbf{x},T)$ ,where $\mathbf{r}$ is inter-class ratio, $T$ the budget,and $\mathbf{x}$ intra-class allocation．To first order:

\mathrm{d}P = \nabla_{\mathbf{r}} f \cdot \mathrm{d}\mathbf{r} + \sum_{c} \nabla_{\mathbf{x}_c} f \cdot \mathrm{d}\mathbf{x}_c + \frac{\partial f}{\partial T} \mathrm{d}T.\tag{4}

The protocol fixes $\mathbf{r},T$ ,and optimizer $g$ ,while admitting the candidate into exactly one class $c$ ．The induced shift $\mathrm{d}\mathbf{x}_c$ is part of the response to a single admission intervention:

\Delta P_{d^{\star}} = f\big(\mathbf{r}, g(\mathcal{D} \cup \{d^{\star}\}), T\big) - f\big(\mathbf{r}, g(\mathcal{D}), T\big)\tag{5}

This captures the total effect of admittingthe dataset—its direct contribution plus budget-preserving re-balancing within its class．

Training Stages

Alignment(8B tokens):High-quality image-caption pairs align visual features with the LLM;only the projector is trainable．
Mixture Validation: Full-parameter fine-tuning on mixed datasets to assess comprehensive capabilities．
Annealing: Short,high-quality refinement stage with learning rate decaying to zero．

Models are assembled from pretrained unimodal-stage components(Qwen3-VL-ViT visual encoder + text-only Qwen3-4B-Instruct language backbone + randomly initialized projector)rather than fine-tuning an existing VLM．

Empirical Validation / Results

4.1 Main Results

Models trained with 80B additional multimodal continued-pretraining tokens are compared against Qwen3-VL-Instruct baselines(Table 1)：

Table ̄1: Comprehensive Performance Comparison

| Models | Params． | MMBench | MME | MMStar | R．WorldQA | MMMU | BLINK | HalBench | AI2D | OCRBen． | InfoVQA | CharXiv | LogicVis | VisuLog | PuzzleVQA | VMME | MVBench | Avg | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | Qwen3-VL-2B | 2B | 77.4 | 72.5 | 56.1 | 64.1 | 49.6 | 55.3 | 45.1 | 75.8 | 82.2 | 58.2 | 51.6 | 35.6 | 24.4 | 13.4 | 47.8 | 50.2 | 53.7 | | Our-1B | 1B | 72.2 | 72.0 | 54.2 | 62.1 | 41.0 | 52.9 | 44.6 | 71.2 | 80.4 | 52..9 | 48..6 | 31..5 | 24..8 | 30..2 | 45..1 | 53..6 |52..3 | | Qwen3-VL-4B | 4B | 83..7 | 82..0 | 66..9 | 70..1 | 60..0 | 64..4 |