How a Major Sports Publisher Cut AI Bot Traffic by 98% Overnight

A Known Agents customer blocked the vast majority of AI bot visits and reduced its CDN usage with Automatic Robots.txt

AI bot visits dropping 98% after Automatic Robots.txt was added

One of our customers, a major sports publisher, wanted to see exactly how AI agents were accessing its content and gain more control over scraping. Within several days of connecting the site to Agent Analytics, the customer learned that automated requests accounted for 40% of all traffic, and half of those requests were AI-related. After the customer deployed Automatic Robots.txt, AI bot visits fell 98% overnight.

40% of All Traffic Was Automated

The customer's team knew automated traffic was reaching the site, but its existing workflow relied on raw server logs, making that traffic time-consuming to isolate, analyze, and visualize.

Using a simple copy-and-paste Fastly configuration, the team connected its site to Agent Analytics and immediately gained visibility into bot activity through the real-time feed. Dashboards broke automated traffic down by bot, category, and page, while AI Chat Referral Tracking revealed how many human visitors ChatGPT, Claude, and other AI assistants sent back to the site.

After several days, the customer had enough data to distinguish the bots it wanted to block from AI assistants and search crawlers that could send visitors back.

A Full-Coverage Robots.txt in 3 Clicks

Next, the customer installed the WordPress plugin, enabled Automatic Robots.txt, and checked three boxes on the plugin's configuration page:

☑️ Block All AI Data Scrapers
☑️ Block All AI Data Providers
☑️ Block All Undocumented AI Agents

Those selections instantly added rules blocking every known bot in the three categories. The rules automatically update as new bots are discovered.

A Known Agents customer's robots.txt with rules blocking AI bots

The customer also tested rules blocking AI Assistants and AI Search Crawlers, but ultimately chose to allow them because they can send human visitors back to the site rather than simply collect content for model training.

98% Less AI Bot Traffic in 24 Hours

Within 24 hours, the customer's AI bot traffic had fallen off a cliff:

Major crawlers including GPTBot, ClaudeBot, Amazonbot, and Meta-ExternalAgent stopped visiting the site almost entirely.
The customer saw no meaningful increase in unidentified automated traffic or human-looking requests, suggesting that the blocked crawlers had not simply switched to less transparent methods.
The team still saw a small amount of seemingly disallowed traffic, including occasional visits from Bytespider and ShapBot, but at extremely low volumes.

Blocking these bots also reduced the customer's CDN usage and infrastructure overhead.

Robots.txt Isn't the Problem, Incomplete Rules Are

Robots.txt is often dismissed as ineffective, but this customer's results align with the global data: the vast majority of AI bots actually do follow the rules. A full-coverage robots.txt is easy to set up and should always be the first line of defense against AI scraping.

Despite how simple this is, our Publisher Robots.txt Leaderboard shows that most major publishers still have incomplete rules. Publishers should close those gaps before turning to more complex and expensive forms of bot protection. You can assess your own coverage with our AI Bot Robots.txt Checker.

Want to uncover and control the AI bot traffic on your site? Request a demo or reach out to us.