Robots.txt Analysis for healthline.com

Status

Found

The robots.txt was successfully fetched

Open Robots.txt ↗

Blocked Bots

124

How many bots are fully blocked by rules in this robots.txt

See All ↓

Missing AI Bots

How many rules are missing from the blocked AI bot types

See All ↓

AI Bot Coverage

21%

49 out of 62 AI bots are still allowed to scrape this website's content

Robots.txt is still highly effective. AI bots follow the rules 97.8% of the time.

Issues & Recommendations

Some AI Data Scrapers Can Still Train on Your Content

You're only blocking 12 of 43 known AI Data Scrapers. The rest can still crawl your site for AI training, so your current rules only provide partial protection.

Fix This →

Some AI Data Providers Can Still Collect Your Content

You're only blocking 1 of 19 known AI Data Providers. The rest can still collect or supply your content to AI companies, leaving a gap in your data-use policy.

Fix This →

Blocking AI Assistants May Reduce Referral Traffic

You're only blocking 2 known AI Assistants. These bots retrieve pages in response to user questions and can cite or link to your site. Keep them blocked only if preventing that access is intentional.

Blocking AI Search Crawlers Reduces AI Search Visibility

You're only blocking 2 known AI Search Crawlers. These bots index and recommend pages in AI search products. Keep them blocked only if you intentionally want to limit that visibility.

Blocking AI Agents Can Turn Away Real Users

You're only blocking 2 known AI Agents. These bots browse and act on behalf of people, so blocking them can prevent users from reading, comparing, or buying from your site.

Get a full-coverage robots.txt with Automatic Robots.txt, then track AI bot visits with Agent Analytics.

124 Blocked Bots

AmazonBuyForMe

AI Agent

Uses an actual web browser to autonomously complete complex tasks on behalf of a human user

NovaAct

AI Agent

Uses an actual web browser to autonomously complete complex tasks on behalf of a human user

amazon-QBusiness

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

Amzn-User

AI Assistant

Fetches website content in response to a user prompt, to include in an AI-generated answer

Diffbot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

Amazonbot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Applebot-Extended

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

bedrockbot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Bytespider

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

CCBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

ClaudeBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

FacebookBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

GPTBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

meta-externalagent

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

omgili

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Timpibot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

webzio-extended

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

amazon-kendra

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

Amzn-SearchBot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

LinkWalker

Intelligence Gatherer

Analyzes web content for brand safety, competitive insights, and ad targeting

TurnitinBot

Intelligence Gatherer

Analyzes web content for brand safety, competitive insights, and ad targeting

uipbot

Intelligence Gatherer

Analyzes web content for brand safety, competitive insights, and ad targeting

HTTrack 3.0

Scraper

Extracts large amounts of web data, often without explicit website permission

Offline Explorer

Scraper

Extracts large amounts of web data, often without explicit website permission

Alexibot

Uncategorized

Not yet assigned a type

Aqua_Products

Uncategorized

Not yet assigned a type

asterias

Uncategorized

Not yet assigned a type

b2w

Uncategorized

Not yet assigned a type

BackDoorBot

Uncategorized

Not yet assigned a type

BlowFish

Uncategorized

Not yet assigned a type

BotALot

Uncategorized

Not yet assigned a type

BotRightHere

Uncategorized

Not yet assigned a type

BuiltBotTough

Uncategorized

Not yet assigned a type

Bullseye

Uncategorized

Not yet assigned a type

BunnySlippers

Uncategorized

Not yet assigned a type

CheeseBot

Uncategorized

Not yet assigned a type

CherryPicker

Uncategorized

Not yet assigned a type

CherryPickerElite

Uncategorized

Not yet assigned a type

CherryPickerSE

Uncategorized

Not yet assigned a type

Copernic

Uncategorized

Not yet assigned a type

CopyRightCheck

Uncategorized

Not yet assigned a type

Cosmos

Uncategorized

Not yet assigned a type

Crescent

Uncategorized

Not yet assigned a type

Crescent Internet ToolPak HTTP OLE Control v.1.0

Uncategorized

Not yet assigned a type

DittoSpyder

Uncategorized

Not yet assigned a type

EmailCollector

Uncategorized

Not yet assigned a type

EmailSiphon

Uncategorized

Not yet assigned a type

EmailWolf

Uncategorized

Not yet assigned a type

EroCrawler

Uncategorized

Not yet assigned a type

ExtractorPro

Uncategorized

Not yet assigned a type

FairAd Client

Uncategorized

Not yet assigned a type

Flaming AttackBot

Uncategorized

Not yet assigned a type

Foobot

Uncategorized

Not yet assigned a type

Gaisbot

Uncategorized

Not yet assigned a type

GetRight

Uncategorized

Not yet assigned a type

Harvest

Uncategorized

Not yet assigned a type

humanlinks

Uncategorized

Not yet assigned a type

InfoNaviRobot

Uncategorized

Not yet assigned a type

Iron33

Uncategorized

Not yet assigned a type

JennyBot

Uncategorized

Not yet assigned a type

Kenjin Spider

Uncategorized

Not yet assigned a type

Keyword Density

Uncategorized

Not yet assigned a type

larbin

Uncategorized

Not yet assigned a type

LexiBot

Uncategorized

Not yet assigned a type

libWeb

Uncategorized

Not yet assigned a type

LinkextractorPro

Uncategorized

Not yet assigned a type

LinkScan

Uncategorized

Not yet assigned a type

LNSpiderguy

Uncategorized

Not yet assigned a type

lwp-trivial

Uncategorized

Not yet assigned a type

Mata Hari

Uncategorized

Not yet assigned a type

MIIxpc

Uncategorized

Not yet assigned a type

Mister PiX

Uncategorized

Not yet assigned a type

moget

Uncategorized

Not yet assigned a type

MSIECrawler

Uncategorized

Not yet assigned a type

NetAnts

Uncategorized

Not yet assigned a type

NICErsPRO

Uncategorized

Not yet assigned a type

NimbleCrawler

Uncategorized

Not yet assigned a type

Openbot

Uncategorized

Not yet assigned a type

openfind

Uncategorized

Not yet assigned a type

Openfind data gatherer

Uncategorized

Not yet assigned a type

Oracle Ultra Search

Uncategorized

Not yet assigned a type

PerMan

Uncategorized

Not yet assigned a type

ProPowerBot

Uncategorized

Not yet assigned a type

ProWebWalker

Uncategorized

Not yet assigned a type

psbot

Uncategorized

Not yet assigned a type

QueryN Metasearch

Uncategorized

Not yet assigned a type

Radiation Retriever 1.1

Uncategorized

Not yet assigned a type

RepoMonkey

Uncategorized

Not yet assigned a type

RepoMonkey Bait & Tackle

Uncategorized

Not yet assigned a type

searchpreview

Uncategorized

Not yet assigned a type

SiteSnagger

Uncategorized

Not yet assigned a type

SpankBot

Uncategorized

Not yet assigned a type

spanner

Uncategorized

Not yet assigned a type

suzuran

Uncategorized

Not yet assigned a type

Szukacz

Uncategorized

Not yet assigned a type

Teleport

Uncategorized

Not yet assigned a type

TeleportPro

Uncategorized

Not yet assigned a type

Telesoft

Uncategorized

Not yet assigned a type

The Intraformant

Uncategorized

Not yet assigned a type

TheNomad

Uncategorized

Not yet assigned a type

toCrawl

Uncategorized

Not yet assigned a type

True_Robot

Uncategorized

Not yet assigned a type

turingos

Uncategorized

Not yet assigned a type

URL Control

Uncategorized

Not yet assigned a type

URL_Spider_Pro

Uncategorized

Not yet assigned a type

URLy Warning

Uncategorized

Not yet assigned a type

VCI

Uncategorized

Not yet assigned a type

VCI WebViewer VCI WebViewer Win32

Uncategorized

Not yet assigned a type

Web Image Collector

Uncategorized

Not yet assigned a type

WebAuto

Uncategorized

Not yet assigned a type

WebBandit

Uncategorized

Not yet assigned a type

WebCapture 2.0

Uncategorized

Not yet assigned a type

WebCopier

Uncategorized

Not yet assigned a type

WebCopier v.2.2

Uncategorized

Not yet assigned a type

WebCopier v3.2a

Uncategorized

Not yet assigned a type

WebEnhancer

Uncategorized

Not yet assigned a type

WebSauger

Uncategorized

Not yet assigned a type

Website Quester

Uncategorized

Not yet assigned a type

WebStripper

Uncategorized

Not yet assigned a type

WebZIP

Uncategorized

Not yet assigned a type

WWW-Collector-E

Uncategorized

Not yet assigned a type

zeus

Uncategorized

Not yet assigned a type

Zeus Link Scout

Uncategorized

Not yet assigned a type

anthropic-ai

Undocumented AI Agent

Crawls websites without disclosing its purpose, collecting data for an unknown AI use case

49 Missing AI Bots

AI2Bot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Ai2Bot-Dolma

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

AIWebIndex

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

ApifyBot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

ApifyWebsiteContentCrawler

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

Bravebot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

Brightbot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

ChatGLM-Spider

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

CloudVertexBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

cohere-training-data-crawler

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Connect your website to see all 49 missing AI bots, block them with Automatic Robots.txt, and track their activity with Agent Analytics.

See Missing AI Bots Request a Demo

Frequently Asked Questions

Does robots.txt even work?

You should always have one as a first line of defense. The data shows that the vast majority of first-party AI Data Scrapers and third-party AI Data Providers actually do follow robots.txt rules.

The few bad actors willing to ignore them are usually small and have limited reach, so they probably aren't much of a threat to your business anyway.

Which AI bots should I block?

Block bots based on what they do, not simply because they use AI. Consider blocking AI Data Scrapers and AI Data Providers if you don't want your content used for model training or resale. You may also want to block general Scrapers and Undocumented AI Agents when their purpose or benefit is unclear.

Usually allow AI Search Crawlers, AI Assistants, and user-directed AI Agents if you want visibility, citations, and potential customer traffic. Keep traditional Search Engine Crawlers allowed unless you want your pages removed from their search results.

Once your rules are in place, use Agent Analytics to identify bots that ignore them. Inspect their location and IP address, then forcefully block them.

How do I keep my robots.txt up to date with new AI bots?

Automatic Robots.txt adds rules for all known bots in the categories you choose and keeps them updated as new bots are discovered.

Does blocking AI bots affect my search rankings?

Blocking AI Data Scrapers and other training crawlers does not directly affect your rankings in traditional search engines. Blocking Search Engine Crawlers such as Googlebot can prevent pages from being indexed and cause them to disappear from search results. Blocking AI Search Crawlers can reduce your visibility in AI search products, even if it does not change your traditional rankings.