Back to News

Study: Non-English languages use 1.5x-5x more AI tokens, creating 'tokenizer tax'

#ai#tokenization#language#cost

A Rest of World study reports that non-English languages can consume 1.5x-5x more AI tokens than the cheapest language, resulting in a 'tokenizer tax' that raises costs and reduces context for users. The study highlights how tokenization inefficiencies disproportionately affect non-English speakers.

Coverage timeline

  1. Techmeme
    Study: non-English languages can consume far more AI tokens, with the most expensive using 1.5x-5x more than the cheapest, resulting in a "tokenizer tax" (Ananya Bhattacharya/Rest of World)

    Ananya Bhattacharya / Rest of World : Study: non-English languages can consume far more AI tokens, with the most expensive using 1.5x-5x more than the cheapest, resulting in a “tokenizer tax” — Some languages require far more tokens for AI models to process, raising costs and reducing context for the people who use them.

Related stories

  • AI efficiency gains limited by human 'verification tax', NYT report finds

    A New York Times report synthesizing economic and corporate research indicates that AI has delivered limited efficiency gains in office work, with a human 'verification tax' emerging as a key bottleneck. Surveys by Workday, Google, MIT, and others detail how AI automation creates new human tasks, such as checking and correcting AI outputs, which offset productivity benefits.

    1 reports · 1 sources

  • OpenAI makes Decisions API generally available with voice, visual modalities

    OpenAI has made its Decisions API, a competitor to Jev, generally available for the first time, according to TechCrunch. The API now includes new voice and visual modalities and is priced at $0.10 per 1 million input tokens with no output-token charges, intensifying competition in the decision-model API market.

    2 reports · 2 sources

  • Extreme Sparse Supervision Improves Reasoning in On-Policy Distillation

    Researchers from Amazon and Duke University found that retaining gradient signals from only one randomly selected token per rollout during on-policy distillation (OPD) significantly improves student model reasoning, challenging the assumption that dense token-level supervision is necessary. Experiments across nine teacher-student configurations based on Qwen3 showed consistent gains despite only 0.025% of tokens receiving supervision.

    1 reports · 1 sources

  • OpenAI releases GPT-6.1 Terra with computer-use focus, API price halved

    OpenAI announced GPT-6.1 Terra, a fast, efficient model emphasizing visual reasoning and computer use, priced at $1 per million input tokens and $5 per million output tokens. The standard API price is 50% lower than GPT-5.6 Terra and GPT-6.1 Sol, and the model is available in ChatGPT Work, Codex, and via API (gpt-6.1-terra) for Plus, Pro, Enterprise, and Edu users. In the Agents' Last Exam benchmark, Terra scored 15 points higher than GPT-5.6 Terra at medium reasoning intensity, with about one-third the cost, and matched GPT-6 Astra's performance at one-tenth the token price.

    4 reports · 4 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe