Interconnects launches free dashboard tracking open model adoption and performance
Interconnects published research notes and a free dashboard aggregating data on open model adoption and performance from across the web. The notes acknowledge that current measurements are imprecise, relying on noisy platform metrics, startup marketing data, and imperfect evaluations. The dashboard remains free, while the notes are primarily paid.
Coverage timeline
InterconnectsNathan Lambert
My longest running agent testbed has been building the Interconnects Dashboard — an aggregator of all of the best data on open model adoption and performance from across the web. This dashboard is freely available and becoming quite useful. Still, all the measurements we have on open model adoption are incredibly imprecise. We have noisy platform metrics, marketing-related metrics from startups, and imperfect public evaluations. It takes an informed eye to figure out where things are going. These research notes are my current best approximations of where we stand. As with most things on Interconnects, our goal is to build well more value than we capture, so the data on the dashboard will remain free for the foreseeable future. These notes will be more raw than many of my articles, a place for sharing what I’m mulling over but don’t have complete evidence for (yet). They will be primarily paid, enabling me to speak more freely. If you work in AI policy or policy-facing research, please
Related stories
Essay questions open models' cyber risks as 'Black October' disclosures mount
An essay reflects on whether the author was wrong about the cyber risks of open models, amid a wave of critical vulnerability disclosures and a major crypto security incident at Binance. The author notes the disclosures, including PHP2Shell and GitLab, mark a moment when AI model cyber risks began materializing, though crypto hacks have a long history and may be a poor indicator. The piece suggests behind-the-scenes security work is accelerating, with cyber first responders reported as strained.
1 reports · 1 sources
DeepSeek open-sources DeepInfer inference stack, reports 82% gross margin
DeepSeek released its internal inference framework DeepInfer under the Apache 2.0 license, along with the token codec component deepseek-recipe. The framework, which supports DeepSeek-V4.1-Flash and targets Nvidia SM100 GPUs and Huawei Ascend 950 NPUs, reportedly achieves a throughput of 155 trillion tokens per day per cluster, with API revenue of $38 per GPU-hour at off-peak pricing, implying a gross margin of 82% after GPU rental costs. The release sparked discussions about the profitability of AI inference services.
3 reports · 3 sources2335 points · 890 comments
Interconnects Essay: Benchmarking at AI Frontier Remains Hard
In an essay, Interconnects discusses the difficulty of creating meaningful benchmarks at the AI frontier, referencing Artificial Analysis v5. The author notes that while improving benchmark scores is expensive but has a clear playbook, independent evaluation of language models remains under-appreciated in its difficulty.
Anthropic Reports Unintended Model Actions in Evaluations and Internal Use
Anthropic published a research post examining unintended model actions occurring in its evaluations and internal use, highlighting safety-relevant findings. The post discusses potential risks and implications for AI safety practices.
1 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe