What is ApifyWebsiteContentCrawler?

Decide how you feel about redistribution. A single crawl from ApifyWebsiteContentCrawler can supply your content to many AI companies for training, search, and retrieval, so allowing it spreads your content across products you have no relationship with. Blocking it costs some AI visibility and nothing in traditional search. For comparison, 2% of the top websites we track already have robots.txt rules for ApifyWebsiteContentCrawler.

Yes. ApifyWebsiteContentCrawler is expected to follow robots.txt rules, so a disallow rule is the right first move. Automatic Robots.txt adds and maintains that rule for you, and Agent Analytics confirms ApifyWebsiteContentCrawler actually follows it.

ApifyWebsiteContentCrawler targets public pages, but at industrial scale. Some providers route requests through large proxy networks that sidestep rate limits and geographic blocks. Content you want kept out of AI systems is safer behind real authentication than behind soft barriers.

One of Apify's customers requested data your site contains, or your pages are part of ApifyWebsiteContentCrawler's standing index. A single fetch can end up serving many downstream AI applications.

Agent Analytics tracks ApifyWebsiteContentCrawler visits and every other known AI agent, crawler, and scraper alongside it, then authenticates each visit against ApifyWebsiteContentCrawler's published verification method. You can also check your server logs for requests whose user agent string contains "ApifyWebsiteContentCrawler". Look for systematic crawling that returns on a schedule. A log match is still not proof, because any bot can claim to be ApifyWebsiteContentCrawler.