Summary (Overview)

• MIRA (Mid-training Rubric Anchoring for Source-Aware Data Selection) is a novel framework for selecting high-quality data from heterogeneous mid-training corpora in LLM development.
• The key innovation is self-anchored rubric discovery: instead of using a fixed global rubric, MIRA automatically discovers source-group-specific quality criteria using a frontier teacher model, then distills these judgments into efficient student scorers.
• MIRA operates in four stages: source clustering, rubric discovery, student distillation, and reliability-aware aggregation with source-aware retention thresholds.
• On code-oriented mid-training with 21 sources in 5 groups, MIRA-Group achieves the best 25B-token macro average (64.20) across nine benchmarks, matching the unfiltered 50B-token full corpus while using only half the tokens.
• The framework addresses a critical mismatch: pretraining filters scale but lack source-specific semantic quality, while post-training filters provide semantic supervision but assume fixed criteria.

---

Introduction and Theoretical Foundation

Background and Motivation

Mid-training has emerged as a distinct stage in modern LLM development, positioned between large-scale pretraining and task-specific post-training. Its data distribution differs from both:

• Pretraining: broad, general web corpora
• Post-training: task-specific, instruction-tuned data
• Mid-training: capability-oriented, curated data (coding, reasoning, long-context, tool use)

Mid-training data forms a hybrid mixture including pretraining-like sources (filtered web documents, code, mathematics, technical text) and structured sources (instruction-style examples, reasoning traces, tool-use logs, multi-turn agent trajectories). This creates a unique data selection challenge.

The Mismatch in Existing Methods

The paper identifies a fundamental gap in existing data selection approaches:

"Pretraining filters scale but do not define source-specific semantic quality, while posttraining filters provide semantic supervision but assume the criteria are already known."

Key Insight

The core theoretical contribution is that rubric construction should be part of data selection itself. Rather than assuming quality criteria are known a priori, MIRA asks: what should be evaluated for each source group? — and derives the answer empirically from the teacher model's actual judgment behavior.

---

Methodology

Overview of the MIRA Pipeline

MIRA consists of four modules:

1. Self-Anchored Rubric Discovery (§3.2)
2. Anchored Judge Distillation (§3.3)
3. Source-Conditioned Reliability Aggregation (§3.4)
4. Source-Preserving Selection (§3.5)

3.2 Self-Anchored Rubric Discovery

Source Clustering: Records from each source are embedded, and sources are clustered by mean content embedding into capability-coherent groups.

Free-form Judging: A frontier teacher (Kimi-K2.6) is presented with sampled records and asked to freely propose quality dimensions, assign scores, and provide reasons — without any pre-specified rubric. This lets the teacher surface the quality concepts it actually uses.

Judgment Clustering and Anchor Extraction: Each free-form response is parsed into (dimension name, reason) pairs. These are embedded and clustered; the point nearest each cluster centroid becomes an anchor dimension. This yields a fixed set of anchor dimensions per group.

3.3 Anchored Judge Distillation

Anchored Teacher Scoring: The teacher re-scores a larger sample (≈2M records total) using the fixed anchor dimensions, producing numerical scores and rationales per dimension.

Student Distillation: Group-specific student scorers (Qwen3.5-35B-A3B-Base, ≈3B active parameters) are fine-tuned to produce scores and rationales for every anchor dimension. A separate student per source group is trained rather than a single universal scorer, because each group's anchors are internally coherent.

3.4 Source-Conditioned Reliability Aggregation

For each source–dimension pair $(s, d)$ with $s \in S_g$ and $d \in \mathcal{D}_g$, the student is evaluated on a validation split and teacher–student agreement statistics (MAE and Spearman correlation) are computed. Pairs below reliability thresholds are flagged, forming a source-conditioned reliability mask:

$$M^{(g)} \in \{0, 1\}^{|S_g| \times |D_g|}$$

The mask is applied post-hoc at aggregation time — only dimensions with $M^{(g)}_{s,d} = 1$ contribute to the overall score, computed as a trimmed mean over surviving dimensions. This avoids prompt-driven score drift and allows low-cost diagnostic updates.

3.5 Source-Preserving Selection

Three selection granularities are defined:

• MIRA-Global: single threshold over the entire scored corpus
• MIRA-Group: retention thresholds within each capability-coherent group (default variant)
• MIRA-Source: thresholds separately within each source

MIRA-Group is the default trade-off: it avoids capability collapse while remaining more stable than per-source thresholding.

---

Empirical Validation / Results

Experimental Setup

• Base model: Qwen2.5-Coder-14B
• Mid-training: Megatron-LM, ~50B tokens, sequence length 128k, global batch size 256, BF16
• Post-training: fixed 400K instruction-following samples (SFT), hyperparameters held constant across conditions
• Teacher: Kimi-K2.6; Students: Qwen3.5-35B-A3B-Base (one per group)
• Corpus: 21 sources, 5 source groups, ~2M teacher-scored records for distillation

Baselines Compared

• No mid-training: Base Model, +SFT only
• Full corpus: Raw Mixture (50B tokens, no filtering)
• 25B-token selection: Random, PPL filtering, DSIR, DataMan, MIRA variants

Main Results (Table 1)

Key findings:

1. MIRA-Group achieves the best macro average (64.20), surpassing all 25B-token baselines and slightly exceeding the unfiltered 50B-token corpus (63.83) at half the token budget.
2. Variant specialization: MIRA-Group leads in code generation (54.53); MIRA-Source leads in Multipl-E (72.84) and SQL (94.38).
3. Random remains competitive because it preserves source diversity — highlighting that generic quality filters can harm capability coverage.

Scorer Analysis (Figure 2)

Length-conditioned analysis reveals:

• PPL drops sharply with token length (strong length dependence)
• DSIR collapses near the long-context region
• DataMan is length-robust but fails on very long records (input-length limit)
• MIRA maintains smooth score profiles across all length ranges, remaining usable around the training cutoff

Reliability Masking (Figure 3)

Radar diagnostics show:

• Student reliability is not uniform across groups (different residual profiles)
• Unreliability is sparse and dimension-specific (concentrated on anchors like A2, A8, A12, A13)
• Post-hoc masking preserves useful signal while removing unstable dimensions

Rubric Space Visualization (Figure 4)

t-SNE analysis shows:

• MIRA rubrics are format-dependent: QA, Text, and Agent rubrics occupy distinct regions
• MIRA covers DataMan's rubric space while being more diverse: 13/14 DataMan dimensions fall inside MIRA's P₉₅ nearest-neighbor distance (0.426), with 5/14 below MIRA's median distance (0.235)

Case Study (Figure 5)

Agent trace analysis shows MIRA captures trajectory-level correctness rather than fluency alone:

• High-scoring traces: valid tool-call payloads, error observation, corrective actions
• Low-scoring traces: concatenated JSON objects, parser errors, repeated invalid calls without recovery

---

Theoretical and Practical Implications

Theoretical Contributions

1. Rubric construction as part of selection: The paper reframes data selection as including the question of what criteria to use, not just how to score. This is a conceptual advance over fixed-rubric approaches.

2. Source-adaptive quality spaces: The finding that different source formats induce fundamentally different quality criteria (Figure 4) challenges the assumption of a universal quality metric.

3. Scalability through distillation: Separating rubric discovery (frontier teacher) from scoring (student models) preserves semantic supervision while achieving full-corpus scalability.

Practical Implications

1. Token efficiency: MIRA-Group matches full-corpus performance at half the token budget, offering significant compute savings for mid-training pipelines.

2. Capability preservation: Source-aware retention prevents capability attrition — the paper shows that global cutoffs drain lower-mean sources first, directly translating into capability loss.

3. Reliability-aware filtering: The post-hoc masking approach provides a low-cost diagnostic framework that can be updated without re-running full-corpus inference.

4. Applicability to heterogeneous corpora: The framework is designed for real-world mid-training mixtures with diverse formats (documents, QA pairs, agent trajectories), which are increasingly common in LLM development.

---

Conclusion

Summary of Contributions

1. Problem framing: The paper identifies and formalizes the unique data selection challenge of heterogeneous mid-training corpora.

2. MIRA framework: A rubric anchoring approach that derives group-specific quality criteria from sampled records using a frontier teacher, avoiding fixed global rubrics.

3. Scalable pipeline: Converts discovered rubrics into structured teacher labels, distills them into group-specific student scorers, and applies reliability-aware aggregation with source-specific retention thresholds.

Key Results

MIRA-Group achieves the best 25B-token macro average (64.20) across nine code-related benchmarks, outperforming PPL, DSIR, DataMan, and Random selection while matching the unfiltered 50B-token corpus at half the token budget.

Limitations and Future Directions

The paper acknowledges that filtering is only one part of mid-training data management. Broader decisions — source discovery, mixture-ratio design, curriculum scheduling, deduplication, contamination control — remain outside scope. Future work could explore:

• Integrating MIRA into broader data-mixture optimization pipelines
• Studying how source-aware quality scores interact with curriculum design
• Investigating interactions with training-time sampling strategies