Back to News

Hugging Face Details 2.8T-Token Indic Pretraining with Open Data Pipeline

#hugging face#indic languages#pretraining#data pipeline

Hugging Face has published details of a large-scale Indic pretraining effort involving 2.8 trillion tokens, alongside an open data generation pipeline. The initiative aims to improve language model performance for Indian languages by providing transparent, reproducible data creation methods.

Coverage timeline

  1. Hugging Face Blog
    Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

Related stories

  • Ai2 releases Olmo-core 3, open training stack for trillion-parameter MoE models

    Ai2 has released Olmo-core 3, a redesigned, fully open training stack for efficiently scaling mixture-of-experts (MoE) models into the trillion-parameter range. The announcement was republished by the Hugging Face Blog.

    2 reports · 2 sources

  • StepFun Unveils Step 5 Preview: 600B MoE, 27B Active, Open-Source Top 2

    StepFun released Step 5 Preview, a 600B-parameter MoE model with 27B active parameters and 1M context, available as an open-weight checkpoint on Hugging Face. In Artificial Analysis evaluations, it scored 44, ranking as the top open-source model after one other, with pricing at $1 per million input tokens and $2.7 per million output tokens. Hands-on tests showed strong agentic capabilities, including autonomously building 3D assets and a LEGO racing game.

    2 reports · 2 sources

  • OpenAI in talks with Anthropic for mutual model stress-testing deal before Hugging Face incident

    According to The Information, before the Hugging Face incident, OpenAI was negotiating a legally binding agreement with Anthropic for the two companies to stress-test each other's models. This move is part of OpenAI's broader rethinking of safety strategies in response to employee and other concerns about AI dangers.

    1 reports · 1 sources

  • OpenAI releases 722 AI-generated math manuscripts, claims on quasi-Riemann and Kakeya

    OpenAI published 722 mathematical manuscripts produced by an unreleased internal frontier model, grouped into 372 result families across 17 fields, including claims on the quasi-Riemann hypothesis, BSD conjecture, and four-dimensional Kakeya conjecture. The company said most results came from a single prompt to a single AI agent, with average compute per result roughly equivalent to three hours of ChatGPT Pro thinking. Mathematicians described the release as overwhelming, noting it could take years to fully understand, and OpenAI withdrew three papers two days later.

    16 reports · 6 sources1334 points · 1520 comments

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe