← Back

Generate Automatic Robots.txt Rules With the REST API

Request

Call this endpoint to generate a new robots.txt with the latest rules. Append the result to your robots.txt file periodically (e.g. once per day, using a cron job).

Endpoint

HTTP Method POST
URL https://api.knownagents.com/robots-txts

Headers

Header Value
Authorization A bearer token with your project's access token (e.g. Bearer YOUR_ACCESS_TOKEN). You can get your project's access token by navigating to the Known Agents Projects page, opening your project, and opening its settings page.
Content-Type Set to application/json.

Body

Field Type Required Description
agent_types string[] Yes An array of agent types you want to block or set a rule for. Allowed agent types include:
  • "AI Agent"
  • "AI Assistant"
  • "AI Coding Agent"
  • "AI Data Provider"
  • "AI Data Scraper"
  • "AI Search Crawler"
  • "Archiver"
  • "Developer Helper"
  • "Fetcher"
  • "Automated Agent"
  • "Intelligence Gatherer"
  • "Scraper"
  • "SEO Crawler"
  • "Search Engine Crawler"
  • "Security Scanner"
  • "Undocumented AI Agent"
  • "Uncategorized"
excluded_agent_ids string[] No An array of agent IDs that should be excluded. You can find an agent's ID in the Metadata section on its page in the Agent Directory. You must be logged in to see the Metadata section.
excluded_operator_ids string[] No An array of operator IDs whose agents should be excluded. You can find an operator's ID in the Metadata section on its page in the Agent Directory. You must be logged in to see the Metadata section.
disallow string No A string specifying which URLs are disallowed. Defaults to / to disallow all URLs.

Example

curl -X POST https://api.knownagents.com/robots-txts \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
        "agent_types": [
            "AI Data Scraper",
            "AI Data Provider",
            "Scraper",
            "SEO Crawler"
        ],
        "disallow": "/"
    }'

Response

The endpoint returns a string in text/plain format.

Example

User-agent: Crawl4AI
Disallow: /

User-agent: DeepSeekBot
Disallow: /

User-agent: CCBot
Disallow: /

...