← Check Another Robots.txt

Robots.txt Analysis for post-gazette.com

This report shows which known AI agents, crawlers, scrapers, and other bots this robots.txt's rules include, and which missing AI bots are still free to scrape the website's content. Missing AI bots include all known first-party AI Data Scrapers and third-party AI Data Providers.
post-gazette.com is ranked #507 on the Publisher Robots.txt Leaderboard.

Status

Found
The robots.txt was successfully fetched
Open Robots.txt

Blocked Bots

10
How many bots are fully blocked by rules in this robots.txt
See All ↓

Missing AI Bots

82
How many rules are missing from the blocked AI bot types
See All ↓

AI Bot Coverage

5%
82 out of 86 AI bots are still allowed to scrape this website's content

Robots.txt is still highly effective. AI bots follow the rules 91.8% of the time.

4 Issues & Opportunities

Get a full-coverage robots.txt with Automatic Robots.txt, then track AI bot visits with Agent Analytics.

10 Blocked Bots

  • AI Assistant
    AI Assistant
    Fetches website content in response to a user prompt, to include in an AI-generated answer
  • AI Data Scraper
    AI Data Scraper
    Downloads website content to include in datasets used for training AI models such as LLMs
  • AI Data Scraper
    AI Data Scraper
    Downloads website content to include in datasets used for training AI models such as LLMs
  • AI Data Scraper
    AI Data Scraper
    Downloads website content to include in datasets used for training AI models such as LLMs
  • AI Data Scraper
    AI Data Scraper
    Downloads website content to include in datasets used for training AI models such as LLMs
  • AI Search Crawler
    AI Search Crawler
    Indexes website content to possibly include as citations in AI-powered search results
  • Archiver
    Archiver
    Captures and stores historical website snapshots for long-term digital preservation
  • Developer Helper
    Developer Helper
    Assists with testing, debugging, and ensuring website functionality
  • Search Engine Crawler
    Search Engine Crawler
    Systematically scans and indexes web pages to include in search results
  • Undocumented AI Agent
    Undocumented AI Agent
    Crawls websites without disclosing its purpose, collecting data for an unknown AI use case

82 Missing AI Bots

  • AI Data Provider
    AI Data Provider
    Crawls websites to supply structured content to AI systems as a third-party service
  • AI Data Scraper
    AI Data Scraper
    Downloads website content to include in datasets used for training AI models such as LLMs
  • AI Data Scraper
    AI Data Scraper
    Downloads website content to include in datasets used for training AI models such as LLMs
  • AI Data Provider
    AI Data Provider
    Crawls websites to supply structured content to AI systems as a third-party service
  • AI Data Provider
    AI Data Provider
    Crawls websites to supply structured content to AI systems as a third-party service
  • AI Data Provider
    AI Data Provider
    Crawls websites to supply structured content to AI systems as a third-party service
  • AI Data Provider
    AI Data Provider
    Crawls websites to supply structured content to AI systems as a third-party service
  • AI Data Provider
    AI Data Provider
    Crawls websites to supply structured content to AI systems as a third-party service
  • AI Data Scraper
    AI Data Scraper
    Downloads website content to include in datasets used for training AI models such as LLMs
  • AI Data Scraper
    AI Data Scraper
    Downloads website content to include in datasets used for training AI models such as LLMs
Connect your website to see all 82 missing AI bots, block them with Automatic Robots.txt, and track their activity with Agent Analytics.

Frequently Asked Questions

Does robots.txt even work?

You should always have one as a first line of defense. The data shows that the vast majority of first-party AI Data Scrapers and third-party AI Data Providers actually do follow robots.txt rules.

The few bad actors willing to ignore them are usually small and have limited reach, so they probably aren't much of a threat to your business anyway.


Which AI bots should I block?

Block bots based on what they do, not simply because they use AI. Consider blocking AI Data Scrapers and AI Data Providers if you don't want your content used for model training or resale. You may also want to block general Scrapers and Undocumented AI Agents when their purpose or benefit is unclear.

Usually allow AI Search Crawlers, AI Assistants, and user-directed AI Agents if you want visibility, citations, and potential customer traffic. Keep traditional Search Engine Crawlers allowed unless you want your pages removed from their search results.

Once your rules are in place, use Agent Analytics to identify bots that ignore them, then reject their requests with Bad Bot Blocking.


How do I keep my robots.txt up to date with new AI bots?

Automatic Robots.txt adds rules for all known bots in the categories you choose and keeps them updated as new bots are discovered.


Does blocking AI bots affect my search rankings?

Blocking AI Data Scrapers and other training crawlers does not directly affect your rankings in traditional search engines. Blocking Search Engine Crawlers such as Googlebot can prevent pages from being indexed and cause them to disappear from search results. Blocking AI Search Crawlers can reduce your visibility in AI search products, even if it does not change your traditional rankings.