OpenAI, Google, Microsoft, Anthropic Back New IETF Standard for AI Scraping Opt-Out
OpenAI, Google, Microsoft, and Anthropic announced they will adopt a new standard that lets publishers opt out of AI scraping across multiple crawlers at once. Proposed by Anthropic technical lead Kevin Kelley within the IETF, the shared policy token will be placed in a website's robots.txt file, allowing publishers to apply their AI training preferences across multiple AI companies instead of setting them individually.
Coverage timeline
The Verge AIEmma Roth
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable . With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually. The shared "policy token" signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website's robots.txt file , which has long signaled which web crawlers can and can't scrape a site - whether for search indexing, archiving, or … Read the full story at The Verge.
Techmeme
Emma Roth / The Verge : OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI across multiple crawlers — The new control will allow publishers to apply their AI training preferences across multiple crawlers.
Related stories
Anthropic and OpenAI Seek Control Over External AI Evaluator Publications
Reuters reports that Anthropic and OpenAI have pushed for substantial control over what external AI evaluators can publish, citing concerns that findings could reveal trade secrets. The companies' efforts to restrict evaluator publications highlight tensions between transparency in AI safety testing and protecting proprietary technology.
1 reports · 1 sources
Anthropic's leaked AI bill draft draws criticism; Amodei defends safety approach
A leaked draft of an AI safety bill from Anthropic has drawn criticism from safety advocates for being weaker than the bipartisan FRONTIER Act, with provisions requiring judicial approval before federal agencies could block a model's release, potentially only if 50 or more people could be killed. In response, Anthropic CEO Dario Amodei defended the company's 'race to the top' approach, citing nine safety decisions with 'substantial costs,' and admitted mishandling the draft bill.
3 reports · 2 sources
Meta's New Terms May Allow Scraping of Business-Associated Websites for AI Training
According to a Nieman Lab report, Meta's updated terms of use, effective October 30, will allow the company to scrape websites associated with business accounts on Facebook, Instagram, and WhatsApp, with no opt-out. Legal experts say this could let Meta override robots.txt blocking instructions to train its AI products, though Meta claims it is merely consolidating commercial terms.
2 reports · 2 sources
Fired OpenAI safety researchers urge halt to AI monitoring work, dispute misconduct claims
Three fired OpenAI safety researchers—Jasmine Wang, Tomek Korbak, and Mikita Balesni—published an open letter urging AI labs to halt work that could impair AI monitoring, saying their dismissals are chilling remaining staff. OpenAI defended the firings, citing an investigation that found they violated policies on handling sensitive information, while the researchers dispute the claims and say they were fired for prioritizing safety over company interests.
7 reports · 5 sources365 points · 228 comments
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe