Summary (Overview)

• Core contribution: The paper introduces Tiny Aya L2-Thinker, a 3.35B multilingual reasoning model that achieves >93% L2 reasoning rate (reasoning in the user's prompt language) across 60 languages and 6 benchmarks, demonstrating that in-language reasoning can be achieved at small scale without sacrificing task performance.
• Key finding: Reasoning is a language-agnostic behavior that can be transferred across typologically diverse languages through careful data mixing, without requiring reasoning supervision in every target language.
• Three pillars of success: (1) broader language coverage improves transfer to unseen languages, (2) a small fraction of multilingual non-reasoning data (20–30%) provides a cheap lever for cross-lingual transfer, and (3) a sufficient English reasoning backbone is essential, especially for difficult math tasks.
• Data mixing beats alternatives: Joint data mixing outperforms sequential adaptation and weight merging, offering the best trade-off between task accuracy and L2 reasoning rate.
• Resource release: The authors release model weights (tiny-aya-l2-thinker, tiny-aya-en-thinker, tiny-aya-base-32K) and multilingual reasoning data to support further research.

Introduction and Theoretical Foundation

The paper addresses the English-centric nature of reasoning in language models. While multilingual models can respond to non-English prompts, they overwhelmingly reason internally in English regardless of the prompt language—a phenomenon the authors term the "multilingual reasoning gap."

Key motivations:

• "Lost in translation" risk: Switching from non-English prompts to English reasoning risks losing intent, nuance, and framing specific to the source language context.
• Knowledge accessibility: Cultural knowledge about specific regions occurs more frequently in the language spoken there (Sahu et al., 2026), so native-language reasoning benefits culturally embedded questions.
• Inspection and evaluation: English-only reasoning limits the ability of non-English speakers to inspect, evaluate, and interact with intermediate reasoning traces—critical in high-stakes domains like healthcare.
• The "multilinguality tax": The paper challenges the assumption that L2 reasoning requires trading off performance.

Theoretical foundation: The work builds on multi-task learning principles (Caruana, 1997; Muennighof et al., 2023), which showed that joint training with mixed data achieves cross-lingual generalization even for languages absent from fine-tuning data. The authors extend this to reasoning language: can multilingual supervision teach a model to condition its reasoning language on the input language?

Methodology

Data Augmentation via Translation

The authors use a "translate-train" approach (Hu et al., 2020), translating English reasoning traces into target languages using command-a-translate and DeepSeek-V3. They treat translated reasoning as a scarce resource, capping each language at 5K samples across 44 diverse languages. Careful filtering removes:

• Prompts with intra-document code-switching
• Trajectories discussing translation or naming target languages
• Prompts with constraints difficult to preserve under translation (exact word counts, formatting requirements)

Multi-Task Post-Training

Three supervision sources are combined:

1. English reasoning (ER): Primary source of task-solving capability
2. Multilingual reasoning (MR): Directly supervises reasoning in target languages
3. Multilingual non-reasoning (NR): Ordinary instruction-following examples without reasoning traces (4.9M samples across 67 languages)

A dual-mode setup is adopted: reasoning examples contain explicit reasoning traces, while non-reasoning examples contain empty thinking blocks with special tokens signaling direct response.

Long-Context Training

The base model's context length is extended from 8K to 32K tokens by resuming cooldown training with a linear learning-rate schedule initialized at $1.25 \times 10^{-4}$, interleaving 8K and 32K context sequences at a 3:1 ratio.

Evaluation Metrics

• Task accuracy: Correctness of final answer per benchmark metric
• L2 reasoning rate: Percentage of samples where reasoning is predominantly in the prompt language (via FastText with GlotLID fallback)
• Repetition rate: Fraction of repeated 4-grams as a doomlooping proxy:

$$\frac{\sum_{g: c_g > 1} (c_g - 1)}{\sum_{g} c_g}.\tag{1}$$

Empirical Validation / Results

Main Results (Table 1)

Key Findings

1. Forcing reasoning language is not a substitute for training: Thinking prefix LF lifts Qwen3.5-4B's L2 rate but falls short of Tiny Aya L2-Thinker, which exceeds it on every benchmark with more stability across languages.

2. In-language reasoning costs minimal accuracy: Switching to L2 reasoning drops accuracy by at most 2–3% on five of six tasks, with the only notable drop on PolyMath (11.1% vs. 18.6%).

3. Long traces signal doomlooping, not deeper reasoning: The 4-gram repetition score rises with mean thinking tokens across models. Qwen3.5-4B sits at the high-token, high-repetition extreme, while Tiny Aya L2-Thinker controls length far better.

4. Low-resource language robustness: Tiny Aya L2-Thinker maintains L2 rates near ceiling on tiers 1–2 (99%) and declines only modestly on lower-resourced languages (94.3% tier 3, 94.5% tier 4), while Magistral-Small-24B collapses (72% → 5%).

Ablation Findings

• Language coverage (§5.1): Expanding from 1-Lang specialists to All-Langs model improves unseen-language L2 reasoning rate monotonically (14→35→60 on MGSM) without interference on seen languages.
• Non-reasoning data (§5.2): A small fraction (10%) raises unseen-language L2 rate from 46% to 89% and task accuracy from 49% to 67% on MGSM. Sweet spot at 20–30%; beyond ~40%, the model starts skipping reasoning.
• English backbone (§5.3): Math benchmarks gain substantially from heavier English data via knowledge transfer; open-ended tasks are largely insensitive to English fraction.

Theoretical and Practical Implications

Theoretical implications:

• Reasoning capability and reasoning language are separable: A model can acquire problem-solving capability from English supervision while learning the language of reasoning through multilingual supervision—challenging the "multilinguality tax" assumption.
• L2 reasoning is a transferable behavioral property: It generalizes to languages without direct reasoning supervision, suggesting reasoning is language-agnostic (as it is for humans).
• The curse of multilinguality does not apply to reasoning: Broader language coverage improves transfer without interference.

Practical implications:

• Accessibility: Enables non-English speakers to inspect and interact with reasoning traces in their own language—critical for high-stakes domains.
• Data efficiency: Requires only a small amount of multilingual reasoning data (<5K samples/language), bringing multilingual reasoning within reach for low-resource languages.
• Deployment: At 3.35B scale, the model is suitable for low-resource deployments while achieving broader coverage (45 vs. 6 languages) than M-Thinker-7B at half the size.

Conclusion

The paper demonstrates that multilingual reasoning need not be built language by language. By treating reasoning language as a transferable behavioral property learned from a small amount of multilingual reasoning data, reinforced through broader multilingual instruction, and generalized to languages without direct reasoning supervision, the authors provide a scalable path toward reasoning systems that are capable across languages and able to reason in the language of their users.

Future directions:

• Reducing dependence on translated reasoning traces (which anchor reasoning style to English templates)
• Human evaluation of reasoning trace quality, fluency, and cultural appropriateness
• Addressing prompt sensitivity (models may revert to English without explicit L2 reasoning instructions)

Limitations acknowledged: translation dependence, automatic evaluation proxies, and prompt sensitivity in deployment contexts with constrained system prompting or mixed-language inputs.