Robots.txt Analysis for scoop.co.nz

Status

Found

The robots.txt was successfully fetched

Open Robots.txt ↗

Blocked Bots

122

How many bots are fully blocked by rules in this robots.txt

See All ↓

Missing AI Bots

How many rules are missing from the blocked AI bot types

See All ↓

AI Bot Coverage

63%

23 out of 62 AI bots are still allowed to scrape this website's content

Robots.txt is still highly effective. AI bots follow the rules 97.8% of the time.

Issues & Recommendations

Some AI Data Scrapers Can Still Train on Your Content

You're only blocking 29 of 43 known AI Data Scrapers. The rest can still crawl your site for AI training, so your current rules only provide partial protection.

Fix This →

Some AI Data Providers Can Still Collect Your Content

You're only blocking 10 of 19 known AI Data Providers. The rest can still collect or supply your content to AI companies, leaving a gap in your data-use policy.

Fix This →

Blocking AI Search Crawlers Reduces AI Search Visibility

You're only blocking 1 known AI Search Crawler. These bots index and recommend pages in AI search products. Keep them blocked only if you intentionally want to limit that visibility.

Get a full-coverage robots.txt with Automatic Robots.txt, then track AI bot visits with Agent Analytics.

122 Blocked Bots

ApifyBot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

ApifyWebsiteContentCrawler

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

Bravebot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

Brightbot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

Diffbot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

ExaBot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

FirecrawlAgent

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

ShapBot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

TavilyBot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

YouBot

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

Ai2Bot-Dolma

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Amazonbot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Applebot-Extended

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Bytespider

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

CCBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

ChatGLM-Spider

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

ClaudeBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

CloudVertexBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

cohere-training-data-crawler

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Cotoyogi

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Datenbank Crawler

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

FacebookBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Google-Extended

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

GoogleOther

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

GPTBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

ICC-Crawler

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

imageSpider

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Kangaroo Bot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

laion-huggingface-processor

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

LCC

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

meta-externalagent

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

netEstate Imprint Crawler

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

omgili

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

PanguBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

SBIntuitionsBot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Spider

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

Timpibot

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

VelenPublicWebCrawler

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

webzio-extended

AI Data Scraper

Downloads website content to include in datasets used for training AI models such as LLMs

PetalBot

AI Search Crawler

Indexes website content to possibly include as citations in AI-powered search results

TikTokSpider

Fetcher

Retrieves web page metadata to power app features like link previews or feeds

aiHitBot

Intelligence Gatherer