OpenAI's Ted Sanders: LLM performance largely explained by model size, version numbers signal release order and improvement magnitude
In a quote shared by Simon Willison, OpenAI's Ted Sanders explains that while LLM performance is multidimensional, model size accounts for much of the variation: larger models tend to be more performant, cost more per token, and recall esoteric facts better. He also clarifies that version numbers like 6.1 indicate release order, relationship to other releases, and magnitude of improvement, rather than a simple linear progression.
Coverage timeline
Simon Willison
In general, LLM performance is incredibly multidimensional. There's no guarantee of a fixed ordering where the model that is best at math is also the model that's best at jokes. Despite that complexity, much of the variation in performance is explained by just a single variable: model size. In general, models that are bigger will (a) be more performant / reliable, (b) cost more per token, and (c) have better recall of esoteric facts. As you note, our models are ordered: Luna < Terra < Sol < Astra [ ... ] These days, version number signifies a few things at once: the order of release, the relationship to other releases, and the magnitude of improvement. So something like 6.1 would indicate it was launched after 5.6, may share some similarities to other 6.1 models in terms of training data / personality (which is why it's not 5.7), and isn't a giant enough leap to merit the headlines and expectations that would come with a 7. — Ted Sanders , explaining OpenAI's model naming conventions T
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe