← Check Another Robots.txt

Robots.txt Analysis for france24.com

This report shows which known AI agents, crawlers, scrapers, and other bots this robots.txt's rules include, and which missing AI bots are still free to scrape the website's content. Missing AI bots include all known first-party AI Data Scrapers and third-party AI Data Providers.

Status

Found
The robots.txt was successfully fetched
Open Robots.txt ↗

Blocked Bots

215
How many bots are fully blocked by rules in this robots.txt
See All ↓

Missing AI Bots

44
How many rules are missing from the blocked AI bot types
See All ↓

AI Bot Coverage

29%
44 out of 62 AI bots are still allowed to scrape this website's content

Robots.txt is still highly effective. AI bots follow the rules 97.8% of the time.

Issues & Recommendations

Some AI Data Scrapers Can Still Train on Your Content
You're only blocking 16 of 43 known AI Data Scrapers. The rest can still crawl your site for AI training, so your current rules only provide partial protection.
Fix This →
Some AI Data Providers Can Still Collect Your Content
You're only blocking 2 of 19 known AI Data Providers. The rest can still collect or supply your content to AI companies, leaving a gap in your data-use policy.
Fix This →
Blocking AI Assistants May Reduce Referral Traffic
You're only blocking 7 known AI Assistants. These bots retrieve pages in response to user questions and can cite or link to your site. Keep them blocked only if preventing that access is intentional.
Blocking AI Search Crawlers Reduces AI Search Visibility
You're only blocking 3 known AI Search Crawlers. These bots index and recommend pages in AI search products. Keep them blocked only if you intentionally want to limit that visibility.

Get a full-coverage robots.txt with Automatic Robots.txt, then track AI bot visits with Agent Analytics.

215 Blocked Bots

ChatGPT-User
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
Claude-User
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
DuckAssistBot
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
MistralAI-User
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
Perplexity-User
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
TongyiBot
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
YiyanBot
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
Claude-Code
AI Coding Agent
AI Coding Agent
Fetches documentation and other resources to help build software
ExaBot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
YouBot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
AI2Bot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Ai2Bot-Dolma
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
CCBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
ClaudeBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
cohere-training-data-crawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
DeepSeekBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
ERNIEBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Google-Extended
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
GPTBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
KimiBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
KimiCrawler
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
MoonshotBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
omgili
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
PanguBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
QwenBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
SBIntuitionsBot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Claude-SearchBot
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
OAI-SearchBot
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
PerplexityBot
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
rogerbot
Developer Helper
Developer Helper
Assists with testing, debugging, and ensuring website functionality
UptimeRobot
Developer Helper
Developer Helper
Assists with testing, debugging, and ensuring website functionality
bitlybot
Fetcher
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
GnowitNewsbot
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
linkfluence
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
LinkWalker
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
Mediatoolkitbot
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
netEstate NE Crawler
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
scoop.it
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
SentiBot
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
trendictionbot
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
008
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
HTTrack
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
HTTrack 3.0
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
Nutch
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
Offline Explorer
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
OpenindexSpider
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
SeznamBot
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
Sogou web spider
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
YisouSpider
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
AhrefsBot
SEO Crawler
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
BLEXBot
SEO Crawler
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
DotBot
SEO Crawler
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
MJ12bot
SEO Crawler
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
SemrushBot
SEO Crawler
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
SEOkicks
SEO Crawler
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
5emeRue
Uncategorized
Uncategorized
Not yet assigned a type
5erue
Uncategorized
Uncategorized
Not yet assigned a type
adequat
Uncategorized
Uncategorized
Not yet assigned a type
adequat-systems
Uncategorized
Uncategorized
Not yet assigned a type
Alexibot
Uncategorized
Uncategorized
Not yet assigned a type
AmiSoftware
Uncategorized
Uncategorized
Not yet assigned a type
Ask n read
Uncategorized
Uncategorized
Not yet assigned a type
asknread.com
Uncategorized
Uncategorized
Not yet assigned a type
asterias
Uncategorized
Uncategorized
Not yet assigned a type
Augure
Uncategorized
Uncategorized
Not yet assigned a type
auramundi
Uncategorized
Uncategorized
Not yet assigned a type
BackDoorBot
Uncategorized
Uncategorized
Not yet assigned a type
BacklinkCrawler
Uncategorized
Uncategorized
Not yet assigned a type
Baidu-YunGuanCe
Uncategorized
Uncategorized
Not yet assigned a type
Black Hole
Uncategorized
Uncategorized
Not yet assigned a type
Blekkobot
Uncategorized
Uncategorized
Not yet assigned a type
BlowFish
Uncategorized
Uncategorized
Not yet assigned a type
BotALot
Uncategorized
Uncategorized
Not yet assigned a type
BuiltBotTough
Uncategorized
Uncategorized
Not yet assigned a type
Bullseye
Uncategorized
Uncategorized
Not yet assigned a type
BunnySlippers
Uncategorized
Uncategorized
Not yet assigned a type
Cegbfeieh
Uncategorized
Uncategorized
Not yet assigned a type
CheeseBot
Uncategorized
Uncategorized
Not yet assigned a type
CherryPicker
Uncategorized
Uncategorized
Not yet assigned a type
CherryPickerElite
Uncategorized
Uncategorized
Not yet assigned a type
CherryPickerSE
Uncategorized
Uncategorized
Not yet assigned a type
Cision
Uncategorized
Uncategorized
Not yet assigned a type
Cliqzbot
Uncategorized
Uncategorized
Not yet assigned a type
coexel
Uncategorized
Uncategorized
Not yet assigned a type
ConveraCrawler
Uncategorized
Uncategorized
Not yet assigned a type
CopyRightCheck
Uncategorized
Uncategorized
Not yet assigned a type
Corporama
Uncategorized
Uncategorized
Not yet assigned a type
Cosmos
Uncategorized
Uncategorized
Not yet assigned a type
Crescent
Uncategorized
Uncategorized
Not yet assigned a type
Crescent Internet ToolPak HTTP OLE Control v.1.0
Uncategorized
Uncategorized
Not yet assigned a type
Digimind
Uncategorized
Uncategorized
Not yet assigned a type
discobot
Uncategorized
Uncategorized
Not yet assigned a type
DittoSpyder
Uncategorized
Uncategorized
Not yet assigned a type
Download Ninja
Uncategorized
Uncategorized
Not yet assigned a type
ellisphere
Uncategorized
Uncategorized
Not yet assigned a type
EmailCollector
Uncategorized
Uncategorized
Not yet assigned a type
EmailSiphon
Uncategorized
Uncategorized
Not yet assigned a type
EmailWolf
Uncategorized
Uncategorized
Not yet assigned a type
EroCrawler
Uncategorized
Uncategorized
Not yet assigned a type
EtaoSpider
Uncategorized
Uncategorized
Not yet assigned a type
ExtractorPro
Uncategorized
Uncategorized
Not yet assigned a type
Flamingo_SearchEngine
Uncategorized
Uncategorized
Not yet assigned a type
Foobot
Uncategorized
Uncategorized
Not yet assigned a type
grub-client
Uncategorized
Uncategorized
Not yet assigned a type
Harvest
Uncategorized
Uncategorized
Not yet assigned a type
humanlinks
Uncategorized
Uncategorized
Not yet assigned a type
InfoNaviRobot
Uncategorized
Uncategorized
Not yet assigned a type
infoseek
Uncategorized
Uncategorized
Not yet assigned a type
JennyBot
Uncategorized
Uncategorized
Not yet assigned a type
Jetbot
Uncategorized
Uncategorized
Not yet assigned a type
JikeSpider
Uncategorized
Uncategorized
Not yet assigned a type
k2spider
Uncategorized
Uncategorized
Not yet assigned a type
kbcrawl
Uncategorized
Uncategorized
Not yet assigned a type
Kenjin Spider
Uncategorized
Uncategorized
Not yet assigned a type
Knowings
Uncategorized
Uncategorized
Not yet assigned a type
LamarkBot
Uncategorized
Uncategorized
Not yet assigned a type
larbin
Uncategorized
Uncategorized
Not yet assigned a type
leadbox
Uncategorized
Uncategorized
Not yet assigned a type
LexiBot
Uncategorized
Uncategorized
Not yet assigned a type
libWeb
Uncategorized
Uncategorized
Not yet assigned a type
LinkextractorPro
Uncategorized
Uncategorized
Not yet assigned a type
linko
Uncategorized
Uncategorized
Not yet assigned a type
LinkScan
Uncategorized
Uncategorized
Not yet assigned a type
lwp-trivial
Uncategorized
Uncategorized
Not yet assigned a type
Mata Hari
Uncategorized
Uncategorized
Not yet assigned a type
Meltawer
Uncategorized
Uncategorized
Not yet assigned a type
Meltwater
Uncategorized
Uncategorized
Not yet assigned a type
mention
Uncategorized
Uncategorized
Not yet assigned a type
MIIxpc
Uncategorized
Uncategorized
Not yet assigned a type
Mister PiX
Uncategorized
Uncategorized
Not yet assigned a type
mlbot
Uncategorized
Uncategorized
Not yet assigned a type
moget
Uncategorized
Uncategorized
Not yet assigned a type
Moreover
Uncategorized
Uncategorized
Not yet assigned a type
MS Search 4.0 Robot
Uncategorized
Uncategorized
Not yet assigned a type
MSIECrawler
Uncategorized
Uncategorized
Not yet assigned a type
mytwip
Uncategorized
Uncategorized
Not yet assigned a type
NaverBot
Uncategorized
Uncategorized
Not yet assigned a type
NetAnts
Uncategorized
Uncategorized
Not yet assigned a type
NetMechanic
Uncategorized
Uncategorized
Not yet assigned a type
Newzbin
Uncategorized
Uncategorized
Not yet assigned a type
NICErsPRO
Uncategorized
Uncategorized
Not yet assigned a type
NPBot
Uncategorized
Uncategorized
Not yet assigned a type
Omgilibot
Uncategorized
Uncategorized
Not yet assigned a type
openfind
Uncategorized
Uncategorized
Not yet assigned a type
opinion-tracker
Uncategorized
Uncategorized
Not yet assigned a type
ProPowerBot
Uncategorized
Uncategorized
Not yet assigned a type
ProWebWalker
Uncategorized
Uncategorized
Not yet assigned a type
proxem
Uncategorized
Uncategorized
Not yet assigned a type
psbot
Uncategorized
Uncategorized
Not yet assigned a type
QueryN Metasearch
Uncategorized
Uncategorized
Not yet assigned a type
Qwam content intelligence
Uncategorized
Uncategorized
Not yet assigned a type
RepoMonkey
Uncategorized
Uncategorized
Not yet assigned a type
Riddler
Uncategorized
Uncategorized
Not yet assigned a type
score3
Uncategorized
Uncategorized
Not yet assigned a type
SearchmetricsBot
Uncategorized
Uncategorized
Not yet assigned a type
SEOkicks-Robot
Uncategorized
Uncategorized
Not yet assigned a type
Sindup
Uncategorized
Uncategorized
Not yet assigned a type
sistrix crawler
Uncategorized
Uncategorized
Not yet assigned a type
SiteBot
Uncategorized
Uncategorized
Not yet assigned a type
sitecheck.internetseer.com
Uncategorized
Uncategorized
Not yet assigned a type
SiteSnagger
Uncategorized
Uncategorized
Not yet assigned a type
SiteSucker
Uncategorized
Uncategorized
Not yet assigned a type
Sosospider
Uncategorized
Uncategorized
Not yet assigned a type
SpankBot
Uncategorized
Uncategorized
Not yet assigned a type
spanner
Uncategorized
Uncategorized
Not yet assigned a type
spotter
Uncategorized
Uncategorized
Not yet assigned a type
SputnikBot
Uncategorized
Uncategorized
Not yet assigned a type
SuperBot
Uncategorized
Uncategorized
Not yet assigned a type
SurveyBot
Uncategorized
Uncategorized
Not yet assigned a type
suzuran
Uncategorized
Uncategorized
Not yet assigned a type
Synthesio
Uncategorized
Uncategorized
Not yet assigned a type
Szukacz
Uncategorized
Uncategorized
Not yet assigned a type
Talkwater
Uncategorized
Uncategorized
Not yet assigned a type
Teleport
Uncategorized
Uncategorized
Not yet assigned a type
TeleportPro
Uncategorized
Uncategorized
Not yet assigned a type
Telesoft
Uncategorized
Uncategorized
Not yet assigned a type
The Intraformant
Uncategorized
Uncategorized
Not yet assigned a type
TheNomad
Uncategorized
Uncategorized
Not yet assigned a type
Titan
Uncategorized
Uncategorized
Not yet assigned a type
toCrawl
Uncategorized
Uncategorized
Not yet assigned a type
trendeo
Uncategorized
Uncategorized
Not yet assigned a type
trendybuzz
Uncategorized
Uncategorized
Not yet assigned a type
True_Robot
Uncategorized
Uncategorized
Not yet assigned a type
turingos
Uncategorized
Uncategorized
Not yet assigned a type
Updownerbot
Uncategorized
Uncategorized
Not yet assigned a type
URLy Warning
Uncategorized
Uncategorized
Not yet assigned a type
VCI
Uncategorized
Uncategorized
Not yet assigned a type
vecteurplus
Uncategorized
Uncategorized
Not yet assigned a type
verticalsearch
Uncategorized
Uncategorized
Not yet assigned a type
vsw
Uncategorized
Uncategorized
Not yet assigned a type
Web Image Collector
Uncategorized
Uncategorized
Not yet assigned a type
WebAuto
Uncategorized
Uncategorized
Not yet assigned a type
WebBandit
Uncategorized
Uncategorized
Not yet assigned a type
WebCopier
Uncategorized
Uncategorized
Not yet assigned a type
WebEnhancer
Uncategorized
Uncategorized
Not yet assigned a type
WebmasterWorldForumBot
Uncategorized
Uncategorized
Not yet assigned a type
WebReaper
Uncategorized
Uncategorized
Not yet assigned a type
WebSauger
Uncategorized
Uncategorized
Not yet assigned a type
Website Quester
Uncategorized
Uncategorized
Not yet assigned a type
WebStripper
Uncategorized
Uncategorized
Not yet assigned a type
WebZIP
Uncategorized
Uncategorized
Not yet assigned a type
winello
Uncategorized
Uncategorized
Not yet assigned a type
WinHTTrack
Uncategorized
Uncategorized
Not yet assigned a type
WiseGuys Robot
Uncategorized
Uncategorized
Not yet assigned a type
WWW-Collector-E
Uncategorized
Uncategorized
Not yet assigned a type
Xenu Link Sleuth
Uncategorized
Uncategorized
Not yet assigned a type
xovibot
Uncategorized
Uncategorized
Not yet assigned a type
Youmag
Uncategorized
Uncategorized
Not yet assigned a type
Zealbot
Uncategorized
Uncategorized
Not yet assigned a type
zeus
Uncategorized
Uncategorized
Not yet assigned a type
Zite
Uncategorized
Uncategorized
Not yet assigned a type
ZyBorg
Uncategorized
Uncategorized
Not yet assigned a type
anthropic-ai
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
Claude-Web
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
cohere-ai
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case

44 Missing AI Bots

AIWebIndex
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Amazonbot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
ApifyBot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
ApifyWebsiteContentCrawler
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Applebot-Extended
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
bedrockbot
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Bravebot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Brightbot
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
Bytespider
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
ChatGLM-Spider
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Connect your website to see all 44 missing AI bots, block them with Automatic Robots.txt, and track their activity with Agent Analytics.
See Missing AI Bots Request a Demo

Frequently Asked Questions

Does robots.txt even work?

You should always have one as a first line of defense. The data shows that the vast majority of first-party AI Data Scrapers and third-party AI Data Providers actually do follow robots.txt rules.

The few bad actors willing to ignore them are usually small and have limited reach, so they probably aren't much of a threat to your business anyway.

Which AI bots should I block?

Block bots based on what they do, not simply because they use AI. Consider blocking AI Data Scrapers and AI Data Providers if you don't want your content used for model training or resale. You may also want to block general Scrapers and Undocumented AI Agents when their purpose or benefit is unclear.

Usually allow AI Search Crawlers, AI Assistants, and user-directed AI Agents if you want visibility, citations, and potential customer traffic. Keep traditional Search Engine Crawlers allowed unless you want your pages removed from their search results.

Once your rules are in place, use Agent Analytics to identify bots that ignore them. Inspect their location and IP address, then forcefully block them.

How do I keep my robots.txt up to date with new AI bots?

Automatic Robots.txt adds rules for all known bots in the categories you choose and keeps them updated as new bots are discovered.

Does blocking AI bots affect my search rankings?

Blocking AI Data Scrapers and other training crawlers does not directly affect your rankings in traditional search engines. Blocking Search Engine Crawlers such as Googlebot can prevent pages from being indexed and cause them to disappear from search results. Blocking AI Search Crawlers can reduce your visibility in AI search products, even if it does not change your traditional rankings.