Back to News

Qwen3.8 27B Addition-in-Words Test Replicated on Local Hardware

#qwen#llm-evaluation#local-inference#reasoning

Simon Willison replicated Colin Frasier's GPT-4o addition-in-words experiment using Qwen3.8-27B on a DGX Spark, with and without reasoning enabled. With reasoning disabled, 30 attempts per combination were run; with reasoning enabled, only one attempt per pair due to longer processing time.

Coverage timeline

  1. Simon Willison

    Research: Qwen3.8 27B addition in words Colin Frasier posted on Bluesky about an experiment he ran over two years ago using GPT-4o to see how well it could "compute the sum but return the answer in words" across increasingly large numbers. Here's the chart he shared of those results: I'm confident GPT-4o didn't cheat and use a calculator, especially since it got so many of the calculations wrong, but I was inspired to run the experiment again on local hardware (a DGX Spark) to explore the effect in a fully controlled environment. I pasted his image into a Codex Remote session (GPT-6 Astra) and had it run the same experiment using Qwen3.8-27B-Q4_K_M.gguf . Here's the result for a run of 30 attempts per combination with reasoning disabled: Then I ran it again with reasoning enabled. This took a lot longer per pair, so instead of running 30 samples per square I ran just one - which results in a much less visually appealing heatmap since each square is either 100% or 0%: It got the right a

Related stories

  • Qwen 3.8 Adopts GPT-5.5 Pro Reasoning Prefills

    A Hacker News post reports that Qwen 3.8 follows the reasoning prefill patterns of GPT-5.5 Pro, suggesting a notable model release or update. The post highlights the adoption of similar reasoning techniques, indicating a trend in model development.

    1 reports · 1 sources234 points · 93 comments
  • GPT-6 Astra in ChatGPT Work generates running routes from OSM data

    Simon Willison tested ChatGPT Work with GPT-6 Astra (Max) to generate 5K and 10K running routes from his home using OpenStreetMap data. The tool worked for 27 minutes and produced the requested routes as embedded visualizations and downloadable GPX and GeoJSON files, but the underlying code was not visible in the UI. Willison criticized this lack of transparency as an anti-feature, noting that after thread compaction, ChatGPT could not provide the Python code it had used.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe