Automatic Robots.txt
Block entire categories of bots with a robots.txt that adds rules for new bots automatically.
# KNOWN AGENTS (updated daily)
User-agent: GPTBot
Disallow: /
User-agent: DeepSeekBot
Disallow: /
User-agent: CCBot
Disallow: /
...
Sign Up With GoogleSet rules for bot categories, not individual bots
Automatic Robots.txt adds a section to your robots.txt file with rules for all of the bots in the categories you select.
Rules for new bots are added automatically
Manual edits are slow, incomplete, and risky
A single typo in your robots.txt can quietly tank your traffic and cost you customers.
Save money on your server bill
Add the generated block alongside your existing directives, then change your category selections whenever your policy changes.
Frequently Asked Questions
Can't bots ignore robots.txt rules?
Technically yes, but a good robots.txt file will solve 90%+ of the problem. You should always use one as your first line of defense.
If a company is small enough to get away with ignoring robots.txt rules, it probably isn't much of a threat anyway, even if it's able to get hold of your data. The vast majority of large companies follow robots.txt rules, including OpenAI, Anthropic, Meta, Amazon, and Microsoft.
The hard part is knowing which of their bots to block. That's where Automatic Robots.txt can help.
Won't other bots just scrape my pages and sell the data?
It's possible, but Automatic Robots.txt also lets you block third-party AI Data Providers and general Scrapers, in addition to first-party AI Data Scrapers. Many of them also follow robots.txt rules.
If a company is small enough to get away with ignoring robots.txt rules, it probably isn't much of a threat anyway, even if it's able to get hold of your data.
You can also use Agent Analytics to monitor suspicious activity and bots that ignore your rules. You can then inspect their attributes, such as location and IP address, and block them with your firewall.
What about bots that don't identify themselves?
Use Agent Analytics to monitor unidentified automated browsers, suspicious activity, and bots that ignore your rules. You can then inspect their attributes, such as location and IP address, and block them with your firewall.
Will this interfere with my existing robots.txt rules?
No. Automatic Robots.txt adds a managed section to your existing robots.txt file, leaving you in full control of every other rule.
How can I enforce these rules?
The WordPress plugin can automatically return an HTTP 403 response to bots that ignore your rules. WordPress VIP customers can use the Enterprise API with WordPress VIP Security Controls to forcefully block misbehaving bots by user agent at the edge.
On other platforms, use Agent Analytics to monitor bots that ignore your rules, then block them with your firewall.
Will Automatic Robots.txt save me money?
Yes. When bots that follow robots.txt rules see a disallow rule, they stop visiting those pages altogether. Fewer requests mean lower bandwidth, compute, logging, and CDN costs.
A firewall acts only after a request is made. An edge firewall can protect your origin server, but it still has to process each attempt.
Is this compatible with my existing firewall (e.g. Cloudflare)?
Yes. Automatic Robots.txt works alongside your existing CDN, firewall, and other server-side protections.
This layered approach is ideal: robots.txt is the first line of defense, telling bots which rules to follow, while your firewall provides a second layer by blocking bots that ignore them.
Robots.txt also saves bandwidth and server costs because bots that follow the rules never make the disallowed requests. A firewall acts only after a request is made. An edge firewall can protect your origin server, but it still has to process each attempt.
Automate your robots.txt for free in under 5 minutes
Use a platform connector or call the API from your backend code
