Robots.txt Analysis for cincinnati.com

Status

Found

The robots.txt was successfully fetched

Open Robots.txt ↗

Blocked Bots

146

How many bots are fully blocked by rules in this robots.txt

See All ↓

Missing AI Bots

How many rules are missing from the blocked AI bot types

See All ↓

AI Bot Coverage

65%

22 out of 62 AI bots are still allowed to scrape this website's content

Robots.txt is still highly effective. AI bots follow the rules 97.8% of the time.

Issues & Recommendations

Some AI Data Scrapers Can Still Train on Your Content

You're only blocking 30 of 43 known AI Data Scrapers. The rest can still crawl your site for AI training, so your current rules only provide partial protection.

Fix This →

Some AI Data Providers Can Still Collect Your Content

You're only blocking 10 of 19 known AI Data Providers. The rest can still collect or supply your content to AI companies, leaving a gap in your data-use policy.

Fix This →

Blocking AI Assistants May Reduce Referral Traffic

You're only blocking 16 known AI Assistants. These bots retrieve pages in response to user questions and can cite or link to your site. Keep them blocked only if preventing that access is intentional.

Blocking AI Search Crawlers Reduces AI Search Visibility

You're only blocking 17 known AI Search Crawlers. These bots index and recommend pages in AI search products. Keep them blocked only if you intentionally want to limit that visibility.

Blocking AI Agents Can Turn Away Real Users

You're only blocking 7 known AI Agents. These bots browse and act on behalf of people, so blocking them can prevent users from reading, comparing, or buying from your site.

Get a full-coverage robots.txt with Automatic Robots.txt, then track AI bot visits with Agent Analytics.

146 Blocked Bots

AmazonBuyForMe

AI Agent

Uses an actual web browser to autonomously complete complex tasks on behalf of a human user

ChatGPT Agent

AI Agent

Uses an actual web browser to autonomously complete complex tasks on behalf of a human user

Google-Agent

AI Agent

Uses an actual web browser to autonomously complete complex tasks on behalf of a human user

GoogleAgent-Mariner

AI Agent

Uses an actual web browser to autonomously complete complex tasks on behalf of a human user

Manus-User

AI Agent

Uses an actual web browser to autonomously complete complex tasks on behalf of a human user

NovaAct

AI Agent

Uses an actual web browser to autonomously complete complex tasks on behalf of a human user

TwinAgent

AI Agent

Uses an actual web browser to autonomously complete complex tasks on behalf of a human user

AI2Bot-DeepResearchEval

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

Amzn-User

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

bigsur.ai

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

ChatGPT-User

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

Claude-User

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

DuckAssistBot

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

Gemini-Deep-Research

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

Google-NotebookLM

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

KlaviyoAIBot

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

LinerBot

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

meta-externalfetcher

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

MistralAI-User

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

Perplexity-User

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

PhindBot

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

Poggio-Citations

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

QualifiedBot

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

Devin

AI Coding Agent

Fetches documentation and other resources to help build software

ApifyBot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

Bravebot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

Brightbot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

Crawlspace

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

Diffbot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

ExaBot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

FirecrawlAgent

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

ShapBot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

TavilyBot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

YouBot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

AI2Bot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Ai2Bot-Dolma

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Amazonbot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Applebot-Extended

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Bytespider

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

CCBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

ChatGLM-Spider

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

ClaudeBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

cohere-training-data-crawler

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Cotoyogi

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Datenbank Crawler

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

DeepSeekBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

FacebookBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Google-Extended

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

GoogleOther

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

GPTBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

ICC-Crawler

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

imageSpider

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Kangaroo Bot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

laion-huggingface-processor

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

LCC

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

meta-externalagent

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

netEstate Imprint Crawler

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

omgili

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

PanguBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

SBIntuitionsBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Spider

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Timpibot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

VelenPublicWebCrawler

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

webzio-extended

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

AddSearchBot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

Amzn-SearchBot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

Anomura

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

Applebot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

atlassian-bot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

Channel3Bot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

Claude-SearchBot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

Cloudflare-AutoRAG

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

Google-CloudVertexBot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

LinkupBot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

meta-webindexer

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

OAI-SearchBot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

PerplexityBot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

PetalBot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

YandexAdditional

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

YandexAdditionalBot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

ZanistaBot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

archive.org_bot

Archiver

Captures and stores historical website snapshots for long-term digital preservation

Arquivo-web-crawler

Archiver

Captures and stores historical website snapshots for long-term digital preservation

ia_archiver

Archiver

Captures and stores historical website snapshots for long-term digital preservation

ia_archiver-web.archive.org

Archiver

Captures and stores historical website snapshots for long-term digital preservation

Turnitin

Archiver

Captures and stores historical website snapshots for long-term digital preservation

WARDBot

Developer Helper

Assists with testing, debugging, and ensuring website functionality

EchoboxBot

Fetcher

Retrieves web page metadata to power app features like link previews or feeds

TikTokSpider

Fetcher

Retrieves web page metadata to power app features like link previews or feeds

aiHitBot

Intelligence Gatherer

Analyzes web content for brand safety, competitive insights, and ad targeting

AudigentAdBot

Intelligence Gatherer

Analyzes web content for brand safety, competitive insights, and ad targeting

AwarioSmartBot

Intelligence Gatherer

Analyzes web content for brand safety, competitive insights, and ad targeting

Echobot Bot

Intelligence Gatherer

Analyzes web content for brand safety, competitive insights, and ad targeting

factset_spyderbot

Intelligence Gatherer

Analyzes web content for brand safety, competitive insights, and ad targeting

ImagesiftBot

Intelligence Gatherer

Analyzes web content for brand safety, competitive insights, and ad targeting

magpie-crawler

Intelligence Gatherer

Analyzes web content for brand safety, competitive insights, and ad targeting

Netvibes

Intelligence Gatherer

Analyzes web content for brand safety, competitive insights, and ad targeting

peer39_crawler

Intelligence Gatherer

Analyzes web content for brand safety, competitive insights, and ad targeting

TurnitinBot

Intelligence Gatherer

Analyzes web content for brand safety, competitive insights, and ad targeting

Scrapy

Scraper

Extracts large amounts of web data, often without explicit website permission

Baiduspider

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

Baiduspider-render

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

coccocbot-web

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

IbouBot

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

Kagibot

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

MojeekBot

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

Neevabot

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

NewsNow

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

SeekportBot

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

SeznamBot

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

Sogou web spider

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

yacybot

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

yandex

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

YandexBot

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

YandexImages

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

YisouSpider

Search Engine Crawler

Systematically scans and indexes web pages to include in search results

ISSCyberRiskCrawler

Security Scanner

Scans websites for security vulnerabilities, threats, and configuration weaknesses

AhrefsBot

SEO Crawler

Analyzes website structure and content to identify SEO improvement opportunities

Barkrowler

SEO Crawler

Analyzes website structure and content to identify SEO improvement opportunities

DataForSeoBot

SEO Crawler

Analyzes website structure and content to identify SEO improvement opportunities

DotBot

SEO Crawler

Analyzes website structure and content to identify SEO improvement opportunities

MJ12bot

SEO Crawler

Analyzes website structure and content to identify SEO improvement opportunities

SemrushBot

SEO Crawler

Analyzes website structure and content to identify SEO improvement opportunities

SemrushBot-OCOB

SEO Crawler

Analyzes website structure and content to identify SEO improvement opportunities

AspiegelBot

Uncategorized

Not yet assigned a type

AwarioRssBot

Uncategorized

Not yet assigned a type

Baiduspider-image

Uncategorized

Not yet assigned a type

Baiduspider-video

Uncategorized

Not yet assigned a type

crawler4j

Uncategorized

Not yet assigned a type

JenkersBot

Uncategorized

Not yet assigned a type

LivelapBot

Uncategorized

Not yet assigned a type

MauiBot

Uncategorized

Not yet assigned a type

Meltwater

Uncategorized

Not yet assigned a type

mojeek

Uncategorized

Not yet assigned a type

MoodleBot

Uncategorized

Not yet assigned a type

NimbleCrawler

Uncategorized

Not yet assigned a type

Omgilibot

Uncategorized

Not yet assigned a type

Poseidon Research Crawler

Uncategorized

Not yet assigned a type

Seekr

Uncategorized

Not yet assigned a type

Sidetrade indexer bot

Uncategorized

Not yet assigned a type

TaraGroup Intelligent Bot

Uncategorized

Not yet assigned a type

anthropic-ai

Undocumented AI Agent

Crawls websites without disclosing its purpose, collecting data for an unknown AI use case

Claude-Web

Undocumented AI Agent

Crawls websites without disclosing its purpose, collecting data for an unknown AI use case

cohere-ai

Undocumented AI Agent

Crawls websites without disclosing its purpose, collecting data for an unknown AI use case

Crawl4AI

Undocumented AI Agent

Crawls websites without disclosing its purpose, collecting data for an unknown AI use case

iaskspider

Undocumented AI Agent

Crawls websites without disclosing its purpose, collecting data for an unknown AI use case

22 Missing AI Bots

AIWebIndex

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

ApifyWebsiteContentCrawler

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

bedrockbot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

CloudVertexBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

CohereBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

CragCrawler

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

Doubaobot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

ERNIEBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

HenkBot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

KimiBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Connect your website to see all 22 missing AI bots, block them with Automatic Robots.txt, and track their activity with Agent Analytics.

See Missing AI Bots Request a Demo

Frequently Asked Questions

Does robots.txt even work?

You should always have one as a first line of defense. The data shows that the vast majority of first-party AI Data Scrapers and third-party AI Data Providers actually do follow robots.txt rules.

The few bad actors willing to ignore them are usually small and have limited reach, so they probably aren't much of a threat to your business anyway.

Which AI bots should I block?

Block bots based on what they do, not simply because they use AI. Consider blocking AI Data Scrapers and AI Data Providers if you don't want your content used for model training or resale. You may also want to block general Scrapers and Undocumented AI Agents when their purpose or benefit is unclear.

Usually allow AI Search Crawlers, AI Assistants, and user-directed AI Agents if you want visibility, citations, and potential customer traffic. Keep traditional Search Engine Crawlers allowed unless you want your pages removed from their search results.

Once your rules are in place, use Agent Analytics to identify bots that ignore them. Inspect their location and IP address, then forcefully block them.

How do I keep my robots.txt up to date with new AI bots?

Automatic Robots.txt adds rules for all known bots in the categories you choose and keeps them updated as new bots are discovered.

Does blocking AI bots affect my search rankings?

Blocking AI Data Scrapers and other training crawlers does not directly affect your rankings in traditional search engines. Blocking Search Engine Crawlers such as Googlebot can prevent pages from being indexed and cause them to disappear from search results. Blocking AI Search Crawlers can reduce your visibility in AI search products, even if it does not change your traditional rankings.