# MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection

> MIRA discovers source-specific quality rubrics via a frontier teacher and distills them into student scorers, matching full-corpus performance at half the token budget.

- **Source:** [arXiv](https://arxiv.org/abs/2605.30288)
- **Published:** 2026-08-29
- **Permalink:** https://picx.dev/p/Ab323h
- **Whiteboard:** https://picx.dev/p/Ab323h/image

## Summary

## Summary (Overview)

- **MIRA** (Mid-training Rubric Anchoring for Source-Aware Data Selection) is a novel framework for selecting high-quality data from heterogeneous mid-training corpora in LLM development.
- The key innovation is **self-anchored rubric discovery**: instead of using a fixed global rubric, MIRA automatically discovers source-group-specific quality criteria using a frontier teacher model, then distills these judgments into efficient student scorers.
- MIRA operates in four stages: source clustering, rubric discovery, student distillation, and reliability-aware aggregation with source-aware retention thresholds.
- On code-oriented mid-training with **21 sources in 5 groups**, MIRA-Group achieves the best 25B-token macro average (64.20) across nine benchmarks, **matching the unfiltered 50B-token full corpus while using only half the tokens**.
- The framework addresses a critical mismatch: pretraining filters scale but lack source-specific semantic quality, while post-training filters provide semantic supervision but assume fixed criteria.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Mid-training has emerged as a distinct stage in modern LLM development, positioned between large-scale pretraining and task-specific post-training. Its data distribution differs from both:

- **Pretraining**: broad, general web corpora
- **Post-training**: task-specific, instruction-tuned data
- **Mid-training**: capability-oriented, curated data (coding, reasoning, long-context, tool use)

Mid-training data forms a **hybrid mixture** including pretraining-like sources (filtered web documents, code, mathematics, technical text) and structured sources (instruction-style examples, reasoning traces, tool-use logs, multi-turn agent trajectories). This creates a unique data selection challenge.

### The Mismatch in Existing Methods

The paper identifies a fundamental gap in existing data selection approaches:

| Approach | Strengths | Weaknesses |
|----------|-----------|------------|
| **Pretraining-oriented** (perplexity, gradients, influence, distribution matching) | Scalable | Implicit signals; no explicit semantic criteria |
| **Post-training-oriented** (human criteria, preference signals, LLM judgments) | Strong semantic supervision | Assume fixed criteria; struggle with heterogeneous formats |

> "Pretraining filters scale but do not define source-specific semantic quality, while posttraining filters provide semantic supervision but assume the criteria are already known."

### Key Insight

The core theoretical contribution is that **rubric construction should be part of data selection itself**. Rather than assuming quality criteria are known a priori, MIRA asks: *what should be evaluated for each source group?* — and derives the answer empirically from the teacher model's actual judgment behavior.

---

## Methodology

### Overview of the MIRA Pipeline

MIRA consists of four modules:

1. **Self-Anchored Rubric Discovery** (§3.2)
2. **Anchored Judge Distillation** (§3.3)
3. **Source-Conditioned Reliability Aggregation** (§3.4)
4. **Source-Preserving Selection** (§3.5)

### 3.2 Self-Anchored Rubric Discovery

**Source Clustering**: Records from each source are embedded, and sources are clustered by mean content embedding into capability-coherent groups.

**Free-form Judging**: A frontier teacher (Kimi-K2.6) is presented with sampled records and asked to freely propose quality dimensions, assign scores, and provide reasons — *without any pre-specified rubric*. This lets the teacher surface the quality concepts it actually uses.

**Judgment Clustering and Anchor Extraction**: Each free-form response is parsed into (dimension name, reason) pairs. These are embedded and clustered; the point nearest each cluster centroid becomes an anchor dimension. This yields a fixed set of anchor dimensions per group.

### 3.3 Anchored Judge Distillation

**Anchored Teacher Scoring**: The teacher re-scores a larger sample (≈2M records total) using the fixed anchor dimensions, producing numerical scores and rationales per dimension.

**Student Distillation**: Group-specific student scorers (Qwen3.5-35B-A3B-Base, ≈3B active parameters) are fine-tuned to produce scores and rationales for every anchor dimension. A **separate student per source group** is trained rather than a single universal scorer, because each group's anchors are internally coherent.

### 3.4 Source-Conditioned Reliability Aggregation

For each source–dimension pair $(s, d)$ with $s \in S_g$ and $d \in \mathcal{D}_g$, the student is evaluated on a validation split and teacher–student agreement statistics (MAE and Spearman correlation) are computed. Pairs below reliability thresholds are flagged, forming a **source-conditioned reliability mask**:

$$M^{(g)} \in \{0, 1\}^{|S_g| \times |D_g|}$$

The mask is applied **post-hoc** at aggregation time — only dimensions with $M^{(g)}_{s,d} = 1$ contribute to the overall score, computed as a **trimmed mean** over surviving dimensions. This avoids prompt-driven score drift and allows low-cost diagnostic updates.

### 3.5 Source-Preserving Selection

Three selection granularities are defined:

- **MIRA-Global**: single threshold over the entire scored corpus
- **MIRA-Group**: retention thresholds within each capability-coherent group (default variant)
- **MIRA-Source**: thresholds separately within each source

MIRA-Group is the default trade-off: it avoids capability collapse while remaining more stable than per-source thresholding.

---

## Empirical Validation / Results

### Experimental Setup

- **Base model**: Qwen2.5-Coder-14B
- **Mid-training**: Megatron-LM, ~50B tokens, sequence length 128k, global batch size 256, BF16
- **Post-training**: fixed 400K instruction-following samples (SFT), hyperparameters held constant across conditions
- **Teacher**: Kimi-K2.6; **Students**: Qwen3.5-35B-A3B-Base (one per group)
- **Corpus**: 21 sources, 5 source groups, ~2M teacher-scored records for distillation

### Baselines Compared

- **No mid-training**: Base Model, +SFT only
- **Full corpus**: Raw Mixture (50B tokens, no filtering)
- **25B-token selection**: Random, PPL filtering, DSIR, DataMan, MIRA variants

### Main Results (Table 1)

| Method | Code Gen Avg. | Multipl-E | SQL Avg. | SWE-M | **Macro Avg.** |
|--------|--------------|-----------|----------|-------|----------------|
| Base Model | 46.89 | 59.14 | 6.29 | 0.33 | 28.16 |
| +SFT | 53.91 | 72.57 | 64.24 | 3.67 | 48.60 |
| Raw Mixture (50B) | 53.71 | 67.42 | 94.18 | 40.00 | 63.83 |
| DSIR | 48.74 | 67.26 | 95.20 | 27.00 | 59.55 |
| PPL | 50.52 | 57.74 | 90.66 | 20.00 | 54.73 |
| Random | 52.71 | 71.44 | 93.79 | 35.00 | 63.23 |
| DataMan | 53.82 | 71.38 | 93.84 | 33.00 | 63.01 |
| **MIRA-Global** | 53.12 | 67.84 | 94.26 | 32.00 | 61.81 |
| **MIRA-Group** | **54.53** | 71.85 | 94.08 | **36.33** | **64.20** |
| **MIRA-Source** | 54.18 | **72.84** | **94.38** | 30.33 | 62.93 |

**Key findings**:
1. **MIRA-Group achieves the best macro average (64.20)**, surpassing all 25B-token baselines and slightly exceeding the unfiltered 50B-token corpus (63.83) at half the token budget.
2. **Variant specialization**: MIRA-Group leads in code generation (54.53); MIRA-Source leads in Multipl-E (72.84) and SQL (94.38).
3. **Random remains competitive** because it preserves source diversity — highlighting that generic quality filters can harm capability coverage.

### Scorer Analysis (Figure 2)

Length-conditioned analysis reveals:
- **PPL** drops sharply with token length (strong length dependence)
- **DSIR** collapses near the long-context region
- **DataMan** is length-robust but fails on very long records (input-length limit)
- **MIRA** maintains smooth score profiles across all length ranges, remaining usable around the training cutoff

### Reliability Masking (Figure 3)

Radar diagnostics show:
- Student reliability is **not uniform across groups** (different residual profiles)
- Unreliability is **sparse and dimension-specific** (concentrated on anchors like A2, A8, A12, A13)
- Post-hoc masking preserves useful signal while removing unstable dimensions

### Rubric Space Visualization (Figure 4)

t-SNE analysis shows:
- MIRA rubrics are **format-dependent**: QA, Text, and Agent rubrics occupy distinct regions
- MIRA **covers DataMan's rubric space** while being more diverse: 13/14 DataMan dimensions fall inside MIRA's P₉₅ nearest-neighbor distance (0.426), with 5/14 below MIRA's median distance (0.235)

### Case Study (Figure 5)

Agent trace analysis shows MIRA captures **trajectory-level correctness** rather than fluency alone:
- High-scoring traces: valid tool-call payloads, error observation, corrective actions
- Low-scoring traces: concatenated JSON objects, parser errors, repeated invalid calls without recovery

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Rubric construction as part of selection**: The paper reframes data selection as including the question of *what criteria to use*, not just *how to score*. This is a conceptual advance over fixed-rubric approaches.

2. **Source-adaptive quality spaces**: The finding that different source formats induce fundamentally different quality criteria (Figure 4) challenges the assumption of a universal quality metric.

3. **Scalability through distillation**: Separating rubric discovery (frontier teacher) from scoring (student models) preserves semantic supervision while achieving full-corpus scalability.

### Practical Implications

1. **Token efficiency**: MIRA-Group matches full-corpus performance at half the token budget, offering significant compute savings for mid-training pipelines.

2. **Capability preservation**: Source-aware retention prevents capability attrition — the paper shows that global cutoffs drain lower-mean sources first, directly translating into capability loss.

3. **Reliability-aware filtering**: The post-hoc masking approach provides a low-cost diagnostic framework that can be updated without re-running full-corpus inference.

4. **Applicability to heterogeneous corpora**: The framework is designed for real-world mid-training mixtures with diverse formats (documents, QA pairs, agent trajectories), which are increasingly common in LLM development.

---

## Conclusion

### Summary of Contributions

1. **Problem framing**: The paper identifies and formalizes the unique data selection challenge of heterogeneous mid-training corpora.

2. **MIRA framework**: A rubric anchoring approach that derives group-specific quality criteria from sampled records using a frontier teacher, avoiding fixed global rubrics.

3. **Scalable pipeline**: Converts discovered rubrics into structured teacher labels, distills them into group-specific student scorers, and applies reliability-aware aggregation with source-specific retention thresholds.

### Key Results

MIRA-Group achieves the best 25B-token macro average (64.20) across nine code-related benchmarks, outperforming PPL, DSIR, DataMan, and Random selection while **matching the unfiltered 50B-token corpus at half the token budget**.

### Limitations and Future Directions

The paper acknowledges that filtering is only one part of mid-training data management. Broader decisions — source discovery, mixture-ratio design, curriculum scheduling, deduplication, contamination control — remain outside scope. Future work could explore:

- Integrating MIRA into broader data-mixture optimization pipelines
- Studying how source-aware quality scores interact with curriculum design
- Investigating interactions with training-time sampling strategies

---

_Markdown view of https://picx.dev/p/Ab323h, served by PicX — AI-generated visual whiteboard summaries of research papers._