Publisher Robots.txt Leaderboard
The vast majority of AI bots respect robots.txt, and it saves money by reducing server traffic. Why are so many publishers overlooking it as a first line of defense?
Improving Your Robots.txt Is a Free, Easy Win
- You can have a full-coverage robots.txt that updates automatically as new AI bots are discovered, for free. Yet most publishers still have incomplete rules that allow AI bots to scrape their content.
- Unlike an expensive firewall, robots.txt will even reduce server cost since it stops bots from visiting your website in the first place.
Don't Bots Ignore Robots.txt?
- Despite selective framing from bot blocking vendors, the data shows that the vast majority of first-party AI Data Scrapers and third-party AI Data Providers actually do follow robots.txt rules. Proposed regulation will drive compliance even higher.
- The few bad actors willing to risk ignoring it are usually small and have limited reach, so they probably aren't much of a threat to your business anyway.
- Setting up a firewall can stop them, but doing so adds a substantial amount of time, cost, and a worse user experience (e.g. slower page loads) for a much smaller gain.
Increasing Coverage Increases the Industry's Leverage
Broad robots.txt coverage for AI bots would also send a clear, public signal: publishers expect their work and infrastructure to be respected. Each publisher that improves their robots.txt strengthens collective leverage in negotiations with AI companies.
How the Leaderboard Works
The leaderboard is updated daily using the latest robots.txts of 1,000 leading publishers. We test each one against every known AI Data Scraper and AI Data Provider, combining them into a single AI Scrapers type. Once a robots.txt has any rules for this type, AI bot coverage is the percentage of its known bots that are blocked. A robots.txt with no rules for this type has 100% coverage. Sites blocking none or using a wildcard block are excluded by default but can be included with the filters.
Robots.txt Works...
...Coverage Is the Problem
Coverage Per Publisher
Want to analyze a different site? Use the AI Bot Robots.txt Checker.