Essay: No Public Commons of Posttraining Data for LLMs
An essay argues that while pretraining data for large language models benefited from a vast public commons of human knowledge, no comparable public commons exists for posttraining data. This asymmetry poses challenges for future LLM development, as posttraining data is not as readily available or curated.
Coverage timeline
Hacker Newslukasschwab
We got extraordinarily lucky with pretraining data. Training large language models required a truly vast corpus of text, and it was just kind of lying around. Not exactly by accident. For the last 50 or so years, humanity has been busily creating and digitizing an enormous public record of human knowledge. Most obviously, Wikipedia’s editors have assembled the most comprehensive encyclopedia humanity has ever produced. A vigorous open source movement created a couple petabytes of software source code. Communities on Stack Exchange spent more than a decade asking questions, arguing about answers, and carefully curating the results. Scientists uploaded their papers to ArXiv. Project Gutenberg—almost as old as the internet itself—produced digital editions of out-of-copyright books. And of course, billions of people made websites, commented, had arguments both petty and profound, uploaded photos and paintings, wrote fiction, and—believe it or not—sometimes went and told lies on the interne
Related stories
Essay Argues Against Using LLMs for Confidence Scoring
An essay published on Hacker News argues that using large language models for confidence scoring is a flawed practice, offering a technically substantive critique. The piece contends that LLMs are not well-suited to provide reliable confidence estimates, challenging a common approach in AI systems.
1 reports · 1 sources223 points · 102 comments
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe