← Check Another Robots.txt
Robots.txt Analysis for slate.com
This report shows which known AI agents, crawlers, scrapers, and other bots this robots.txt's rules include, and which missing AI bots are still free to scrape the website's content. Missing AI bots include all known first-party AI Data Scrapers and third-party AI Data Providers.
Status
Found
The robots.txt was successfully fetched
Open Robots.txt ↗
Blocked Bots
64
How many bots are fully blocked by rules in this robots.txt
See All ↓
Missing AI Bots
19
How many rules are missing from the blocked AI bot types
See All ↓
AI Bot Coverage
69%
19 out of 62 AI bots are still allowed to scrape this website's content
Robots.txt is still highly effective. AI bots follow the rules 97.8% of the time.
Issues & Recommendations
You're Not Blocking Any AI Data Providers
You're only blocking AI Data Scrapers, but no AI Data Providers. Data providers can still collect and supply your content to AI companies. If your goal is to prevent AI training or resale, apply your policy to both categories.
Get a full-coverage robots.txt with Automatic Robots.txt, then track AI bot visits with Agent Analytics.
64 Blocked Bots
AI2Bot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Ai2Bot-Dolma
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Amazonbot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Applebot-Extended
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
bedrockbot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Bytespider
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
CCBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
ChatGLM-Spider
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
ClaudeBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
CloudVertexBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
cohere-training-data-crawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
CohereBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Cotoyogi
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Datenbank Crawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
DeepSeekBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Doubaobot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
ERNIEBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
FacebookBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Google-Extended
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
GoogleOther
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
GPTBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
ICC-Crawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
imageSpider
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Kangaroo Bot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
KimiBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
KimiCrawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
laion-huggingface-processor
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
LCC
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
meta-externalagent
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
micro-crawl
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
MistralBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
MoonshotBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
MoonSpider
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
netEstate Imprint Crawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
omgili
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
PanguBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
QwenBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
SalamandraVLM
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
SBIntuitionsBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Spider
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Timpibot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
VelenPublicWebCrawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
webzio-extended
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
008
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
AdsTxtCrawler
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
Dataprovider.com
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
dcrawl
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
HelloworkJobPostingBot
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
HTTrack
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
HTTrack 3.0
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
IndeedJobBot
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
KrawlerBot
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
MetaInspector
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
MetaJobBot
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
newspaper
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
Nutch
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
nyt_scraping
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
Offline Explorer
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
OpenindexSpider
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
Potions
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
Scrapy
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
ServerHunterSpider
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
StatsDroneBot
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
Thinkbot
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
19 Missing AI Bots
AIWebIndex
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
ApifyBot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
ApifyWebsiteContentCrawler
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Bravebot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Brightbot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
CragCrawler
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Crawlspace
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Diffbot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
ExaBot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
FirecrawlAgent
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Frequently Asked Questions
Does robots.txt even work?
You should always have one as a first line of defense. The data shows that the vast majority of first-party AI Data Scrapers and third-party AI Data Providers actually do follow robots.txt rules.
The few bad actors willing to ignore them are usually small and have limited reach, so they probably aren't much of a threat to your business anyway.
Which AI bots should I block?
Block bots based on what they do, not simply because they use AI. Consider blocking AI Data Scrapers and AI Data Providers if you don't want your content used for model training or resale. You may also want to block general Scrapers and Undocumented AI Agents when their purpose or benefit is unclear.
Usually allow AI Search Crawlers, AI Assistants, and user-directed AI Agents if you want visibility, citations, and potential customer traffic. Keep traditional Search Engine Crawlers allowed unless you want your pages removed from their search results.
Once your rules are in place, use Agent Analytics to identify bots that ignore them. Inspect their location and IP address, then forcefully block them.
How do I keep my robots.txt up to date with new AI bots?
Automatic Robots.txt adds rules for all known bots in the categories you choose and keeps them updated as new bots are discovered.
Does blocking AI bots affect my search rankings?
Blocking AI Data Scrapers and other training crawlers does not directly affect your rankings in traditional search engines. Blocking Search Engine Crawlers such as Googlebot can prevent pages from being indexed and cause them to disappear from search results. Blocking AI Search Crawlers can reduce your visibility in AI search products, even if it does not change your traditional rankings.