← Back

Generate Automatic Robots.txt Rules With the REST API

Overview

This guide shows you how to use the Automatic Robots.txt REST API. Please contact us if you need help.

Request

Call this endpoint to generate a new robots.txt with the latest rules. Append the result to your robots.txt file periodically (e.g. once per day, using a cron job).

URL
URL https://api.knownagents.com/robots-txts
HTTP Method POST
Headers
Authorization A bearer token with your project's access token (e.g. Bearer YOUR_ACCESS_TOKEN). You can get your project's access token by navigating to the Known Agents Projects page, opening your project, and opening its settings page.
Content-Type Set to application/json.
Body
Field Type Required Description
agent_types string[] Yes An array of agent types you want to block or set a rule for. Allowed agent types include:
  • AI Agent
  • AI Assistant
  • AI Coding Agent
  • AI Data Provider
  • AI Data Scraper
  • AI Search Crawler
  • Archiver
  • Developer Helper
  • Fetcher
  • Automated Agent
  • Intelligence Gatherer
  • Scraper
  • SEO Crawler
  • Search Engine Crawler
  • Security Scanner
  • Undocumented AI Agent
  • Uncategorized
disallow string No A string specifying which URLs are disallowed. Defaults to / to disallow all URLs.

Example

curl -X POST https://api.knownagents.com/robots-txts \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
        "agent_types": [
            "AI Data Scraper",
            "AI Data Provider",
            "Scraper",
            "SEO Crawler"
        ],
        "disallow": "/"
    }'

Response

The endpoint returns a string in text/plain format.

Example

User-agent: Crawl4AI
Disallow: /

User-agent: DeepSeekBot
Disallow: /

User-agent: CCBot
Disallow: /

...