Set Up Automatic Robots.txt for Your Node.js Backend

Step 1: Install the NPM Package

Download the package from NPM using the command line.

npm install @knownagents/sdk

Step 2: Initialize the Client

In your code, create an instance of KnownAgents with your project's access token.

import { KnownAgents } from "@knownagents/sdk"

const knownAgents = new KnownAgents("YOUR_ACCESS_TOKEN")

• Navigate to the Known Agents Projects page and open your project
• Copy your access token from the Settings page
• Back in your code, swap in your access token where it says YOUR_ACCESS_TOKEN

Step 3: Block Agents by Category

Use the generateRobotsTxt method. Select which AgentTypes you want to block, and a string specifying which URLs are disallowed (e.g. "/" to disallow all paths). Allowed agent types include:

• AI Agent
• AI Assistant
• AI Coding Agent
• AI Data Provider
• AI Data Scraper
• AI Search Crawler
• Archiver
• Developer Helper
• Fetcher
• Automated Agent
• Intelligence Gatherer
• Scraper
• SEO Crawler
• Search Engine Crawler
• Security Scanner
• Undocumented AI Agent
• Uncategorized

const robotsTXT = await knownAgents.generateRobotsTxt([
    AgentType.AIDataScraper,
    AgentType.AIDataProvider,
    AgentType.Scraper,
    AgentType.SEOCrawler
], "/")

The return value is a plain text robots.txt string.

Step 4: Serve Your Robots.txt

Generate and cache a robotsTXT periodically (e.g. once per day, using a cron job). Then, serve it from your website's /robots.txt endpoint.