Robots.txt Analysis for nytimes.com

You should always have one as a first line of defense. The data shows that the vast majority of first-party AI Data Scrapers and third-party AI Data Providers actually do follow robots.txt rules.

The few bad actors willing to ignore them are usually small and have limited reach, so they probably aren't much of a threat to your business anyway.

Block bots based on what they do, not simply because they use AI. Consider blocking AI Data Scrapers and AI Data Providers if you don't want your content used for model training or resale. You may also want to block general Scrapers and Undocumented AI Agents when their purpose or benefit is unclear.

Usually allow AI Search Crawlers, AI Assistants, and user-directed AI Agents if you want visibility, citations, and potential customer traffic. Keep traditional Search Engine Crawlers allowed unless you want your pages removed from their search results.

Once your rules are in place, use Agent Analytics to identify bots that ignore them. Inspect their location and IP address, then forcefully block them.

Automatic Robots.txt adds rules for all known bots in the categories you choose and keeps them updated as new bots are discovered.

Blocking AI Data Scrapers and other training crawlers does not directly affect your rankings in traditional search engines. Blocking Search Engine Crawlers such as Googlebot can prevent pages from being indexed and cause them to disappear from search results. Blocking AI Search Crawlers can reduce your visibility in AI search products, even if it does not change your traditional rankings.