Summary (Overview)

• KletterMix is a 725B-token German pretraining and annealing corpus constructed by translating ClimbMix, a state-of-the-art English pretraining mixture, into German while preserving document boundaries, metadata, source structure, and topical diversity.
• The paper introduces a scalable translation pipeline featuring length-aware batching, contextualized chunking with cross-chunk context windows, dynamic target-side budgeting, and shard-wise parallel execution.
• Quality assessment combines COMETKiwi reference-free quality estimation on a stratified pilot subset with a gradient-boosted proxy model that scores the full corpus using target-only features (GlotLID language signals, text-shape statistics), enabling scalable filtering without source rehydration.
• Controlled training ablations at 0.6B parameters (matched pretraining and annealing) and 7B parameters (language-mixture annealing) show KletterMix achieves measurable improvements on German downstream evaluations, particularly on HellaSwag and ARC-C, while English retention remains nearly unchanged.
• The work demonstrates that carefully curated translated data can transfer not just German surface form but useful mixture structure and reasoning capabilities, complementing—not replacing—native German web corpora.

---

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses a central asymmetry in language model development: while English benefits from large, diverse, and well-documented pretraining corpora, German-language resources remain comparatively immature—often derived from noisy web crawls, embedded as subsets of multilingual corpora, or released with limited documentation and validation through controlled training experiments.

The authors argue that strong German model behavior cannot be assumed to emerge from English-centric data alone due to:

• German morphology (compounding, inflection)
• Capitalization conventions
• Regional variation across German-speaking countries
• Domain-specific register differences

Theoretical Approach

Rather than relying exclusively on native German web crawling, the paper explores a complementary route: transferring the curation decisions, source diversity, and mixture structure of a strong English pretraining corpus into German through high-quality machine translation. This is framed as a dataset-construction problem rather than simple data augmentation, acknowledging risks including:

• Translationese artifacts
• Semantic drift
• Source-language bias
• Length pathologies
• Failures on long or specialized documents

The theoretical foundation draws on recent work showing that pretraining data quality depends not only on scale but on source composition, filtering choices, documentation, and validation through model training (The Pile, ROOTS, Dolma, DCLM, ClimbMix).

---

Methodology

Pipeline Architecture

The KletterMix pipeline consists of five stages:

1. Source-record normalization: Preserves document identifiers, source text, and inherited metadata (source cluster, source location, approximate length).

2. Length-aware routing: Documents are assigned to context buckets (short, medium, long, overflow) based on approximate source length to optimize translation efficiency.

3. Document-preserving translation:

   • Documents fitting the source budget are translated in a single pass
   • Longer documents are chunked: segmented into sentences, greedily packed into source chunks up to a fixed token budget
   • For chunked documents, the prompt for chunk $t$ includes a truncated window of the German translation of chunk $t-1$ for discourse continuity and terminology consistency

4. Scalable shard-wise execution: Multiple workers process disjoint subsets with incremental writes, intermediate attempt files, and resumption capabilities.

5. Post-hoc quality estimation: COMETKiwi scoring on a stratified pilot subset, followed by proxy-model training for full-corpus annotation.

Dynamic Target-Side Budgeting

The maximum target-side generation length is derived from source chunk length:

$$\ell_{\mathrm{tgt}}^{\mathrm{max}} = \min\left(L_{\mathrm{max}}, \left\lceil \alpha \ell_{\mathrm{src}} + \beta \right\rceil\right)$$

where $L_{\mathrm{max}}$ is the maximum generation budget and $\alpha, \beta$ allow moderate target-side expansion.

Proxy-Based Quality Annotation

A gradient-boosted regression model predicts COMETKiwi scores using only target-side features:

• GlotLID features: top label is German Latin (deu_Latn), normalized German-Latin probability, clipped logit transform, top-1/top-2 probability margin, script from top predicted label
• Text-shape features: length, lexical diversity, token repetition, average token length, character-composition ratios

The proxy is validated on a disjoint 18,275-document split, showing strong agreement with COMETKiwi.

Dataset Variants

Three proxy-filtered 12B-token training splits are constructed using thresholds:

$$\hat{q}_{\mathrm{proxy}} \geq 0.50, \quad \hat{q}_{\mathrm{proxy}} \geq 0.55, \quad \hat{q}_{\mathrm{proxy}} \geq 0.60$$

---

Empirical Validation / Results

Corpus-Level Diagnostics

• Document length distribution (Fig. 3a): Heavy-tailed profile typical of heterogeneous web mixtures, with most documents below 10k tokens and a long tail beyond 20k tokens.
• Bucket-level length consistency (Fig. 3b): Long lower tails reveal unusually short translations in long-context buckets, flagged as audit signals for potential truncation or dropped content.
• Cluster-level quality (Fig. 3c): Proxy-score quantiles are broadly consistent between the 12B-token subset and full corpus across inherited source clusters.

0.6B Parameter Pretraining Results

Table 1: Downstream 5-shot accuracy on German evaluations under matched 12B-token training conditions.

Key findings:

• KletterMix reaches lower training and validation loss than FineWeb2-DE and GermanWeb throughout training
• The strongest gains concentrate on HellaSwag and ARC-C (grounded event continuation and compositional science-style reasoning)
• Annealing FineWeb2-DE on KletterMix yields +1.1 points over the source checkpoint and +1.8 points over GermanWeb annealing on Core Avg.

7B Parameter Language-Mixture Annealing

Five runs anneal the OLMo 3 7B Stage 1 checkpoint for ~12B tokens with KletterMix supplying 0%, 5%, 10%, 15%, or 20% of tokens (ClimbMix supplies the remainder):

Replacing 20% of the annealing data with KletterMix improves the German four-task mean by 4.5 percentage points while English changes by only +0.3 points.

---

Theoretical and Practical Implications

Theoretical Significance

• Reasoning transfer through data curation: The results suggest that translated data can transfer more than surface form—it can transfer mixture structure and reasoning capabilities. KletterMix is strongest where coherent document structure and dense explanatory content matter most (HellaSwag, ARC-C), not uniformly across all tasks.
• Complementary role of translated data: KletterMix is not positioned as a replacement for native German data but as a complement. GermanWeb remains strongest on MMLU; FineWeb2-DE remains competitive on PIQA.
• Quality estimation methodology: The target-only proxy model demonstrates that scalable corpus-level quality signals can be built without source rehydration, making the approach practical for large-scale releases.

Practical Implications

• Late-stage language adaptation: The annealing experiments show KletterMix is useful not only as an initialization corpus but as a late-stage sharpening component for event coherence and science-style reasoning.
• Filtering nuance: Proxy scores are useful for ranking data under a fixed budget but not as a universal quality law—MMLU does not improve monotonically with stricter filtering.
• Reusable artifact: Because each German document preserves source identifiers, metadata, clusters, and length buckets, KletterMix supports controlled studies of translation-based data curation.

Limitations

• Inherits topical, cultural, geographic, stylistic, and licensing biases from English sources
• Machine translation may introduce translationese, semantic drift, unnatural style, or inconsistent terminology
• Each training condition has one run; both regimes use ~12B-token budgets; the 7B experiment uses a single starting checkpoint
• Proxy scores are not substitutes for human evaluation or source-aware quality assessment

---

Conclusion

KletterMix demonstrates that careful translation, corpus documentation, and empirical validation constitute a practical path toward stronger non-English pretraining data. The key takeaways are:

1. Document-preserving translation of a high-quality English mixture can produce a German corpus with the scale and diversity of modern pretraining datasets while enabling direct English–German comparison.

2. Controlled training ablations at two scales (0.6B and 7B parameters) show measurable downstream improvements, particularly in reasoning-intensive tasks, without meaningful English degradation.

3. Target-only proxy models provide scalable quality estimation that enables corpus-level filtering and documentation, though they should complement rather than replace source-aware evaluation.

Future Directions

• Stronger filtering targeting translation failures not captured by current diagnostics (URL-only documents, boilerplate, duplicated content, subtle semantic drift)
• Manual audits and broader benchmark evaluations to characterize naturalness and factual preservation
• Extension to other languages (French, Italian, Spanish) to enable systematic comparisons of when translated pretraining mixtures complement native-language web corpora
• Larger models, additional random seeds, and broader downstream suites to strengthen the empirical evidence base