Edu-QuRating: Multi-Dimensional Educational Data Curation with Distilled Pairwise Judgements

Summary (Overview)

• Multi-dimensional educational scoring: Edu-QuRating extends QuRating's pairwise preference distillation to produce reusable single-text scorers across 20 educational dimensions organized into three rubric families: six core educational criteria (Core-Ed), seven student-facing foundational-literacy criteria (FL-Student), and seven teacher-facing foundational-literacy criteria (FL-Teacher).

• High-fidelity preference distillation: The best Edu-QuRater (Gemma-3-4B-PT base) recovers held-out GPT-4.1-mini pairwise judgements with mean accuracy of 0.917 across six criteria, with both base models (Sheared-LLaMA-1.3B and Gemma-3-4B-PT) exceeding 0.86 accuracy on every criterion.

• Improved pre-training via corpus filtering: Applying Edu-QuRaters to score 322.25M FineWeb-Edu-Fortified documents, filtered pre-training mixtures achieved higher observed aggregate accuracy across nine benchmarks than the FineWeb-Edu baseline in matched single-run comparisons, with gains concentrated in specific tasks (ARC-CF and HellaSwag showing the clearest improvements of roughly 3–5 accuracy points).

• Effective GRPO reward modeling: Edu-QuRater scores used as reward terms for GRPO post-training produced responses preferred over the Qwen3-4B base model on both pedagogical quality (81.08% win rate when combined with answer-structure rewards) and instruction following (68.24%).

• Scalable and efficient: A probability-sampling shortcut for LLM judge preferences reduces judge calls by a factor of 20 with mean absolute error below 0.03, and 200k pairwise examples provide a practical training scale for distillation.

Introduction and Theoretical Foundation

Background and Motivation

Data curation has become a central component in language-model training, with broad web filters, deduplication, and mixture design capable of changing model performance as much as architectural choices—especially for smaller models. Educational filtering has been particularly beneficial: FineWeb-Edu demonstrates that selecting education-oriented web text improves performance on knowledge- and reasoning-heavy benchmarks relative to the same architecture trained with unfiltered web data.

However, the authors argue that the question "is this document educational?" is only the beginning. Once a corpus has been filtered for educational content, remaining documents still vary along dimensions that matter for learning:

"A page can be engaging but inaccurate, accurate but badly sequenced, useful for secondary students but inaccessible to primary readers, or broadly educational without supporting foundational literacy."

Treating these cases as a single scalar educational score makes data selection hard to inspect and hard to adapt to concrete instructional goals.

Theoretical Foundations

Bradley–Terry pairwise comparison model: The methodological foundation is the classical Bradley–Terry model, where each item has a latent score, and the probability of preferring one item over another is a logistic function of the score difference. QuRating extends this by replacing fixed item scores with transformer-predicted scores, enabling single-text scoring at scale.

Educational criteria: The rubric design draws on established educational research:

• The Simple View of Reading (Gough and Tunmer, 1986) models skilled reading as the product of decoding and linguistic comprehension
• Scarborough's (2001) reading rope decomposes reading into interacting word-recognition and language-comprehension strands
• The National Reading Panel (2000) and Castles et al. (2018) identify phonemic awareness, phonics, fluency, vocabulary, and comprehension as core components
• The GEEAP reading report (Alvarez Marinelli et al., 2025), synthesizing over 120 studies from low- and middle-income countries, adds oral language and writing, concluding that children do not learn to read incidentally—each component must be taught explicitly and systematically

Cognitive load and learning theory: Pedagogical structure rewards writing a novice can follow, with material sequenced in small steps, worked examples, and minimal extraneous detail (Sweller, 1988; Rosenshine, 2012). Lesson engagement rewards attention hooks and relatable contexts (Hidi and Renninger, 2006), while noting that engaging additions can harm learning when they add extraneous load (Rey, 2012).

Methodology

Pipeline Overview

The Edu-QuRating pipeline (Figure 1) consists of five stages:

1. Sampling: Sample text from an education-rich web corpus (FineWeb-Edu-Fortified, 200k documents sub-sampled proportionally from 95 Common Crawl subsets)
2. Pairwise LLM judging: An LLM judge (GPT-4.1-mini) compares text pairs under criterion-specific rubrics, producing soft pairwise preference labels
3. Distillation: A sequence-classification model is trained with a neural Bradley–Terry objective so that score differences predict judge preferences
4. Corpus scoring: The trained Edu-QuRater scores the full corpus one document at a time
5. Downstream application: Scores are used for pre-training data filtering or GRPO reward modeling

Pairwise Preference Distillation

For each criterion $c$, a text $x_i$ is assigned a score $s_i^{(c)} = f_\theta^{(c)}(x_i)$. Given scores $s_i^{(c)}$ and $s_j^{(c)}$, the implied preference probability is:

$$\hat{p}_{ij}^{(c)} = \sigma\left(s_j^{(c)} - s_i^{(c)}\right),$$

where $\sigma$ is the logistic sigmoid. Training minimizes binary cross-entropy between $\hat{p}_{ij}^{(c)}$ and the LLM-derived soft label $y_{ij}^{(c)} \in [0,1]$.

Efficiency optimization: Rather than repeating many LLM-judge comparisons on the same text pair, the authors read the judge model's relative probabilities for constrained output tokens A and B directly in a single inference pass. This matched a repeated-sampling estimate with mean absolute error below 0.03 while reducing judge calls by a factor of 20.

Base models compared: Sheared-LLaMA-1.3B (following original QuRating) and Gemma-3-4B-PT, trained on 512-token excerpts for two epochs with a 0.1 warmup fraction.

Educational Rubric Families

Core-Ed (6 dimensions): Overall educational level, Primary-level suitability, Secondary-level suitability, Factual accuracy, Lesson engagement, and Pedagogical structure. The three level dimensions are treated as metadata labels rather than quality dimensions, since "a text teaches effectively only when its demands meet the learner's current knowledge."

FL-Student (7 dimensions): Oral language and vocabulary, Phonological awareness, Systematic phonics, Reading fluency, Reading comprehension, Writing and expression, and Engagement and relevance—evaluating text as practice material for beginner readers.

FL-Teacher (7 dimensions): Oral language/vocabulary instruction, Phonological-awareness instruction, Systematic phonics, Reading fluency, Reading comprehension, Writing/encoding, and Pedagogical quality—evaluating instructional material for educators, modeled on the GEEAP report's six components of evidence-based reading instruction.

Pre-training Experiment Setup

Following the Smol Training Playbook ablation configuration, models were trained over 45B tokens with:

• 10% FineMath-3Plus and 20% Stack-Edu-Python fixed across all conditions
• The remaining 70% web-text component varied: FineWeb-Edu (baseline), Edu-QuRating 50-50-50 filtered, stricter-pedagogy filtered, or 35%/35% mixture of stricter-pedagogy data and DataComp-LM (DCLM)

The 50-50-50 rule keeps documents scoring above the median in all three quality dimensions (Factual Accuracy, Lesson Engagement, Pedagogical Structure), retaining 16.12% of scored rows. Corpus scoring used an orchestrator/worker pipeline on 32 H200 GPUs, taking ~72 hours (~2300 GPU hours) at ~40 rows/second per GPU.

GRPO Post-training Setup

Using Qwen3-4B as base model with 1008 teacher-task examples focused on foundational literacy (grades 0-3), four reward configurations were tested:

1. Answer-structure reward only (LLM-judge scoring against synthetic high-quality outputs)
2. Edu-QuRater rewards only (Core-Ed or FL-Teacher scores)
3. Combined Edu-QuRater + answer-structure rewards

Empirical Validation / Results

Distillation Accuracy

Both base models exceed 0.86 accuracy on every criterion for recovering held-out GPT-4.1-mini pairwise preferences. Mean accuracy rises from 0.895 (Sheared-LLaMA-1.3B) to 0.917 (Gemma-3-4B-PT), with Gemma improving accuracy uniformly across every criterion.

Scaling Behavior

Validation loss decreases sharply from 20k to 200k pairwise examples:

200k examples provide a practical training scale, reaching the low-loss region without requiring substantially more LLM-labelled comparisons.

Score Distribution Validity

On ~15,800 external educational materials with standardized annotations:

• Overall Education Level dimension correlates highly with education level metadata (Pearson r = 0.62)
• Primary-level Suitability shows a peak around primary education levels (r = 0.64)
• A linear model combining all six dimensions achieves cross-validated $R^2$ = 0.449 (vs. 0.387–0.405 for the strongest single-dimension models)
• A notable bimodal distribution of factual accuracy scores in preschool materials may reflect fantastical/fairy-tale content

Pre-training Results

At 30k steps, all Edu-QuRating-based mixtures reach higher observed aggregate means than the FineWeb-Edu baseline:

Task-level analysis shows gains concentrated in ARC-CF and HellaSwag (roughly 3–5 accuracy points), with smaller improvements in OpenBookQA, BoolQ, and WinoGrande. MMLU-family improvements are close to zero, and CommonsenseQA is mixed.

GRPO Post-training Results

The Edu-QuRater-only condition shows over-optimization of the proxy reward without directly constraining prompt adherence. Combining both reward types produces the best results on both metrics.

Theoretical and Practical Implications

Theoretical Contributions

1. Multi-dimensionality of educational quality: Edu-QuRating demonstrates that educational value is not a single scalar property but decomposes into independently varying dimensions (accuracy, structure, engagement, level suitability) that can be quantified and used separately.

2. Distillation feasibility: The work shows that pairwise LLM preferences under complex, education-specific rubrics can be reliably distilled into small sequence-classification models, making large-scale scoring computationally feasible.

3. Task-specific gains: Pre-training improvements are concentrated in particular tasks rather than uniform across benchmarks, suggesting that different filtering criteria may optimize for different downstream capabilities.

Practical Implications

1. Educational content curation: Edu-QuRaters support both quality assessment (factual accuracy, pedagogical structure, engagement) and scalable metadata labeling (learner-level suitability, foundational-literacy dimensions), enabling indexing, retrieval, recommendation, and dataset assembly.

2. Global educational context: The approach is particularly relevant for low- and middle-income countries where content needs to suit local curricula, languages of instruction, and classroom conditions. The student- and teacher-facing literacy rubrics offer curation around specific instructional uses.

3. Cost-effective reward modeling: Edu-QuRater rewards can be computed without reference answers, making them potentially useful where high-quality examples are costly to produce.

4. Complementarity with existing filters: Edu-QuRater dimensions show modest correlations with FineWeb-Edu scores (ranging from 0.042 to 0.326), and a linear model combining all six dimensions achieves cross-validated $R^2$ = 0.228, indicating the dimensions capture part of the educational-quality signal while decomposing it into non-redundant components.

Conclusion

Edu-QuRating adapts pairwise preference distillation to multi-dimensional educational data curation, defining three rubric families (Core-Ed, FL-Student, FL-Teacher) with 20 output dimensions. The trained Edu-QuRaters accurately recover held-out LLM pairwise preferences (mean accuracy 0.917 with Gemma-3-4B-PT), and the approach scales to scoring 322.25M documents.

The downstream applications demonstrate practical utility: Edu-QuRating-filtered pre-training mixtures reach higher observed aggregate accuracy than the FineWeb-Edu baseline, and Edu-QuRater rewards improve both pedagogical quality and instruction following in GRPO post-training when combined with answer-structure rewards.

Limitations

• Scorers inherit the coverage of the upstream corpus and cannot compensate for missing languages, curricula, or genres
• Pre-training experiments use one run per mixture due to resource constraints, preventing claims about the most effective possible mixture
• GRPO evaluation is relatively small-scale, and pedagogical criteria for evaluation were designed to overlap with FL-Teacher rubrics
• Real-world classroom benefits were not directly tested

Future Directions

• Educator assessments of selected materials and generated responses
• Studies of learner outcomes to demonstrate practical value across teaching contexts
• Adapting scoring criteria with educators familiar with local curricula and classroom conditions
• Validated mappings from scores to labels for indexing, retrieval, and recommendation systems
• Broader testing of whether GRPO improvements generalize beyond the FL-Teacher framework