# Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoning

> Tiny Aya L2-Thinker achieves over 93% in-language reasoning across 60 languages at 3.35B scale, proving reasoning language is transferable without sacrificing task accuracy.

- **Source:** [arXiv](https://arxiv.org/abs/2609.10445)
- **Published:** 2026-09-26
- **Permalink:** https://picx.dev/p/9YYUIE
- **Whiteboard:** https://picx.dev/p/9YYUIE/image

## Summary

## Summary (Overview)

- **Core contribution**: The paper introduces **Tiny Aya L2-Thinker**, a 3.35B multilingual reasoning model that achieves >93% L2 reasoning rate (reasoning in the user's prompt language) across 60 languages and 6 benchmarks, demonstrating that in-language reasoning can be achieved at small scale without sacrificing task performance.
- **Key finding**: Reasoning is a **language-agnostic behavior** that can be transferred across typologically diverse languages through careful data mixing, without requiring reasoning supervision in every target language.
- **Three pillars of success**: (1) broader language coverage improves transfer to unseen languages, (2) a small fraction of multilingual non-reasoning data (20–30%) provides a cheap lever for cross-lingual transfer, and (3) a sufficient English reasoning backbone is essential, especially for difficult math tasks.
- **Data mixing beats alternatives**: Joint data mixing outperforms sequential adaptation and weight merging, offering the best trade-off between task accuracy and L2 reasoning rate.
- **Resource release**: The authors release model weights (tiny-aya-l2-thinker, tiny-aya-en-thinker, tiny-aya-base-32K) and multilingual reasoning data to support further research.

## Introduction and Theoretical Foundation

The paper addresses the **English-centric nature of reasoning in language models**. While multilingual models can respond to non-English prompts, they overwhelmingly reason internally in English regardless of the prompt language—a phenomenon the authors term the "multilingual reasoning gap."

**Key motivations**:
- **"Lost in translation" risk**: Switching from non-English prompts to English reasoning risks losing intent, nuance, and framing specific to the source language context.
- **Knowledge accessibility**: Cultural knowledge about specific regions occurs more frequently in the language spoken there (Sahu et al., 2026), so native-language reasoning benefits culturally embedded questions.
- **Inspection and evaluation**: English-only reasoning limits the ability of non-English speakers to inspect, evaluate, and interact with intermediate reasoning traces—critical in high-stakes domains like healthcare.
- **The "multilinguality tax"**: The paper challenges the assumption that L2 reasoning requires trading off performance.

**Theoretical foundation**: The work builds on multi-task learning principles (Caruana, 1997; Muennighof et al., 2023), which showed that joint training with mixed data achieves cross-lingual generalization even for languages absent from fine-tuning data. The authors extend this to reasoning language: can multilingual supervision teach a model to condition its reasoning language on the input language?

## Methodology

### Data Augmentation via Translation
The authors use a **"translate-train"** approach (Hu et al., 2020), translating English reasoning traces into target languages using command-a-translate and DeepSeek-V3. They treat translated reasoning as a **scarce resource**, capping each language at 5K samples across 44 diverse languages. Careful filtering removes:
- Prompts with intra-document code-switching
- Trajectories discussing translation or naming target languages
- Prompts with constraints difficult to preserve under translation (exact word counts, formatting requirements)

### Multi-Task Post-Training
Three supervision sources are combined:
1. **English reasoning (ER)**: Primary source of task-solving capability
2. **Multilingual reasoning (MR)**: Directly supervises reasoning in target languages
3. **Multilingual non-reasoning (NR)**: Ordinary instruction-following examples without reasoning traces (4.9M samples across 67 languages)

A **dual-mode setup** is adopted: reasoning examples contain explicit reasoning traces, while non-reasoning examples contain empty thinking blocks with special tokens signaling direct response.

### Long-Context Training
The base model's context length is extended from 8K to 32K tokens by resuming cooldown training with a linear learning-rate schedule initialized at $1.25 \times 10^{-4}$, interleaving 8K and 32K context sequences at a 3:1 ratio.

### Evaluation Metrics
- **Task accuracy**: Correctness of final answer per benchmark metric
- **L2 reasoning rate**: Percentage of samples where reasoning is predominantly in the prompt language (via FastText with GlotLID fallback)
- **Repetition rate**: Fraction of repeated 4-grams as a doomlooping proxy:

$$
\frac{\sum_{g: c_g > 1} (c_g - 1)}{\sum_{g} c_g}.\tag{1}
$$

## Empirical Validation / Results

### Main Results (Table 1)

| Model | MGSM Acc/L2% | PolyMath Acc/L2% | MIST-OEG Acc/L2% | Marco-Bench-MIF Acc/L2% | GlobalPIQA Acc/L2% | Macaron-MCQ Acc/L2% |
|---|---|---|---|---|---|---|
| **Tiny Aya En-Thinker** (Other) | 70.8 / 0.8 | 18.6 / 0.1 | 85.3 / 10.5 | 52.8 / 7.9 | 72.3 / 28.4 | 42.6 / 15.2 |
| **Qwen3.5-4B** (Other) | 52.2 / 31.0 | 39.8 / 0.2 | 79.9 / 15.2 | 23.8 / 31.4 | 78.6 / 7.9 | 37.0 / 12.8 |
| **Tiny Aya L2-Thinker** (Other) | **68.0 / 96.5** | 11.1 / **94.9** | **85.8 / 95.2** | **50.7 / 96.8** | 70.7 / **98.3** | **39.8 / 93.8** |
| Qwen3.5-4B (user prefix LF) | 51.2 / 33.9 | **40.3** / 0.2 | 81.5 / 29.4 | 21.2 / 33.3 | **78.1** / 12.2 | 30.2 / 13.5 |
| Qwen3.5-4B (thinking prefix LF) | 50.6 / 94.3 | 27.6 / 87.6 | 67.7 / 93.3 | 33.8 / 82.3 | 77.0 / 94.7 | 37.7 / 91.1 |
| M-Thinker-7B | 38.0 / 87.8 | 32.6 / 88.0 | 30.0 / 96.5 | 28.2 / 94.1 | 59.2 / 92.2 | 32.7 / 87.7 |
| Magistral-Small-24B | 64.2 / 33.6 | 15.9 / 39.4 | 80.8 / 60.4 | 50.9 / 48.9 | 77.1 / 49.0 | 49.6 / 14.7 |

### Key Findings

1. **Forcing reasoning language is not a substitute for training**: Thinking prefix LF lifts Qwen3.5-4B's L2 rate but falls short of Tiny Aya L2-Thinker, which exceeds it on every benchmark with more stability across languages.

2. **In-language reasoning costs minimal accuracy**: Switching to L2 reasoning drops accuracy by at most 2–3% on five of six tasks, with the only notable drop on PolyMath (11.1% vs. 18.6%).

3. **Long traces signal doomlooping, not deeper reasoning**: The 4-gram repetition score rises with mean thinking tokens across models. Qwen3.5-4B sits at the high-token, high-repetition extreme, while Tiny Aya L2-Thinker controls length far better.

4. **Low-resource language robustness**: Tiny Aya L2-Thinker maintains L2 rates near ceiling on tiers 1–2 (99%) and declines only modestly on lower-resourced languages (94.3% tier 3, 94.5% tier 4), while Magistral-Small-24B collapses (72% → 5%).

### Ablation Findings

- **Language coverage (§5.1)**: Expanding from 1-Lang specialists to All-Langs model improves unseen-language L2 reasoning rate monotonically (14→35→60 on MGSM) without interference on seen languages.
- **Non-reasoning data (§5.2)**: A small fraction (10%) raises unseen-language L2 rate from 46% to 89% and task accuracy from 49% to 67% on MGSM. Sweet spot at 20–30%; beyond ~40%, the model starts skipping reasoning.
- **English backbone (§5.3)**: Math benchmarks gain substantially from heavier English data via knowledge transfer; open-ended tasks are largely insensitive to English fraction.

## Theoretical and Practical Implications

**Theoretical implications**:
- **Reasoning capability and reasoning language are separable**: A model can acquire problem-solving capability from English supervision while learning the language of reasoning through multilingual supervision—challenging the "multilinguality tax" assumption.
- **L2 reasoning is a transferable behavioral property**: It generalizes to languages without direct reasoning supervision, suggesting reasoning is language-agnostic (as it is for humans).
- **The curse of multilinguality does not apply to reasoning**: Broader language coverage improves transfer without interference.

**Practical implications**:
- **Accessibility**: Enables non-English speakers to inspect and interact with reasoning traces in their own language—critical for high-stakes domains.
- **Data efficiency**: Requires only a small amount of multilingual reasoning data (<5K samples/language), bringing multilingual reasoning within reach for low-resource languages.
- **Deployment**: At 3.35B scale, the model is suitable for low-resource deployments while achieving broader coverage (45 vs. 6 languages) than M-Thinker-7B at half the size.

## Conclusion

The paper demonstrates that **multilingual reasoning need not be built language by language**. By treating reasoning language as a transferable behavioral property learned from a small amount of multilingual reasoning data, reinforced through broader multilingual instruction, and generalized to languages without direct reasoning supervision, the authors provide a scalable path toward reasoning systems that are capable across languages and able to reason in the language of their users.

**Future directions**:
- Reducing dependence on translated reasoning traces (which anchor reasoning style to English templates)
- Human evaluation of reasoning trace quality, fluency, and cultural appropriateness
- Addressing prompt sensitivity (models may revert to English without explicit L2 reasoning instructions)

**Limitations** acknowledged: translation dependence, automatic evaluation proxies, and prompt sensitivity in deployment contexts with constrained system prompting or mixed-language inputs.

---

_Markdown view of https://picx.dev/p/9YYUIE, served by PicX — AI-generated visual whiteboard summaries of research papers._