← Check Another Robots.txt
Robots.txt Analysis for golf.com
This report shows which known AI agents, crawlers, scrapers, and other bots this robots.txt's rules include, and which missing AI bots are still free to scrape the website's content. Missing AI bots include all known first-party AI Data Scrapers and third-party AI Data Providers.
Status
Found
The robots.txt was successfully fetched
Open Robots.txt ↗
Blocked Bots
72
How many bots are fully blocked by rules in this robots.txt
See All ↓
Missing AI Bots
0
How many rules are missing from the blocked AI bot types
See All ↓
AI Bot Coverage
100%
The rules fully cover the AI bot types they're blocking
Robots.txt is still highly effective. AI bots follow the rules 97.8% of the time.
Issues & Recommendations
No issues detected
Get a full-coverage robots.txt with Automatic Robots.txt, then track AI bot visits with Agent Analytics.
72 Blocked Bots
AIWebIndex
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
ApifyBot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
ApifyWebsiteContentCrawler
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Bravebot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Brightbot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
CragCrawler
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Crawlspace
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Diffbot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
ExaBot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
FirecrawlAgent
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
HenkBot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Mozilla-Tabstack
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Querit-SearchBot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
QueritBot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
ShapBot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
TavilyBot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Terra Cotta
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
TerraCotta
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
YouBot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
AI2Bot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Ai2Bot-Dolma
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Amazonbot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Applebot-Extended
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
bedrockbot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Bytespider
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
CCBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
ChatGLM-Spider
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
ClaudeBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
CloudVertexBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
cohere-training-data-crawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
CohereBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Cotoyogi
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Datenbank Crawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
DeepSeekBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Doubaobot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
ERNIEBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
FacebookBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Google-Extended
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
GoogleOther
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
GPTBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
ICC-Crawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
imageSpider
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Kangaroo Bot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
KimiBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
KimiCrawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
laion-huggingface-processor
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
LCC
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
meta-externalagent
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
micro-crawl
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
MistralBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
MoonshotBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
MoonSpider
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
netEstate Imprint Crawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
omgili
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
PanguBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
QwenBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
SalamandraVLM
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
SBIntuitionsBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Spider
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Timpibot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
VelenPublicWebCrawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
webzio-extended
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
anthropic-ai
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
Aranet-SearchBot
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
Claude-Web
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
cohere-ai
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
Crawl4AI
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
iAskBot
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
iaskspider
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
KunatoCrawler
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
WRTNBot
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
ZenphonyBot
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
0 Missing AI Bots
No AI bots are missing
Frequently Asked Questions
Does robots.txt even work?
You should always have one as a first line of defense. The data shows that the vast majority of first-party AI Data Scrapers and third-party AI Data Providers actually do follow robots.txt rules.
The few bad actors willing to ignore them are usually small and have limited reach, so they probably aren't much of a threat to your business anyway.
Which AI bots should I block?
Block bots based on what they do, not simply because they use AI. Consider blocking AI Data Scrapers and AI Data Providers if you don't want your content used for model training or resale. You may also want to block general Scrapers and Undocumented AI Agents when their purpose or benefit is unclear.
Usually allow AI Search Crawlers, AI Assistants, and user-directed AI Agents if you want visibility, citations, and potential customer traffic. Keep traditional Search Engine Crawlers allowed unless you want your pages removed from their search results.
Once your rules are in place, use Agent Analytics to identify bots that ignore them. Inspect their location and IP address, then forcefully block them.
How do I keep my robots.txt up to date with new AI bots?
Automatic Robots.txt adds rules for all known bots in the categories you choose and keeps them updated as new bots are discovered.
Does blocking AI bots affect my search rankings?
Blocking AI Data Scrapers and other training crawlers does not directly affect your rankings in traditional search engines. Blocking Search Engine Crawlers such as Googlebot can prevent pages from being indexed and cause them to disappear from search results. Blocking AI Search Crawlers can reduce your visibility in AI search products, even if it does not change your traditional rankings.