# Known Agents > Known Agents, formerly Dark Visitors, helps website owners measure AI agent and bot activity, protect content from unwanted scraping, reduce infrastructure costs, and grow human traffic from AI recommendations. Its products combine traffic analytics, agent identification and verification, automatic robots.txt rules, and bot blocking integrations. Known Agents maintains a public directory of agents, crawlers, and scrapers and publishes research on their behavior. Website owners use this knowledge and their own traffic data to support useful agents, restrict unwanted access, and make informed decisions about content distribution and licensing. The canonical website is https://knownagents.com. The directory, research, product pages, and documentation are public; customer reports and project settings require sign-in. Refer to the linked pages for current agent details, measurements, plan limits, and integration requirements. Important context: - **Traffic:** Analytics integrations send website request and response data to power both Agent Analytics and AI Chat Referral Tracking. Agent visits are automated requests; AI referrals are observed human visits attributed to AI platforms. - **Measurement:** Citation counts are estimates inferred from AI agents retrieving content. Public research describes observed trends across participating websites; its methodology explains the scope and limitations. - **Identity:** Identification recognizes an agent; verification checks its claimed identity using supported methods such as IP matching or HTTP message signatures. Some agents cannot be verified, and an unidentified request may still be automated. - **Access control:** Automatic Robots.txt keeps rules current for selected bot categories alongside existing directives. Cooperative bots follow those rules; rejecting requests requires a separate blocking integration, whose capabilities depend on the platform. - **API access:** The REST API at https://api.knownagents.com uses project access tokens in the `Authorization: Bearer YOUR_ACCESS_TOKEN` header. Analytics requests submit events; identification requests return results for your own code to act on. Access and usage limits depend on the plan. ## Start Here - [About Known Agents](https://knownagents.com/about): The company's mission to build trust between websites and agents, how its products work, and answers for website owners and agent operators. - [Documentation](https://knownagents.com/docs): Installation and API guides for measuring bot traffic and AI referrals, maintaining robots.txt rules, blocking unwanted bots, and identifying incoming agents. - [Pricing](https://knownagents.com/pricing): Compare plans by event allowance, websites, reporting, API access, and team features; explains how events are counted and what happens at usage limits. - [Publisher Guide](https://knownagents.com/publishers): How publishers can protect content, reduce CDN costs, support licensing negotiations with traffic data, and grow AI referrals; includes setup instructions and Enterprise trials. ## Agent Directory and Research - [Agent Directory](https://knownagents.com/agents): Look up agents, crawlers, and scrapers to understand their purpose and operator. Profiles include user agent strings, robots.txt guidance, and observed traffic and blocking trends. - [The Agentic Web Index](https://knownagents.com/insights): Daily research on bot traffic, AI activity, robots.txt effectiveness, impersonation, and AI referrals across participating websites, with methodology and guidance for citing the data. - [AI Bot Robots.txt Checker](https://knownagents.com/tools/robots-txt-checker): Check a domain's allowed and blocked bots, find missing AI training bots in its robots.txt rules, and get recommendations for improving coverage. - [Publisher Robots.txt Leaderboard](https://knownagents.com/publisher-robots-txt-leaderboard): Compare leading publishers by robots.txt coverage of AI training scrapers and data providers, then inspect each domain's rules and missing bots. - [Submit an Agent](https://knownagents.com/submit-an-agent): Submit an agent's user agent string, operator, and documentation for a directory listing that helps website owners recognize it and understand its purpose. ## Products - [Agent Analytics](https://knownagents.com/products/agent-analytics): See which bots visit which pages, track changes and traffic spikes, and diagnose errors, slow responses, spoofed identities, and robots.txt violations. Export reports or analyze them through connected AI assistants. - [AI Chat Referral Tracking](https://knownagents.com/products/ai-chat-referral-tracking): Measure human traffic from ChatGPT, Claude, Gemini, and other AI platforms by source, page, country, and time. Use referral trends and estimated citation click-through rates to guide content strategy. - [Automatic Robots.txt](https://knownagents.com/products/automatic-robots-txt): Keep rules for selected bot categories current as new agents are discovered, helping protect content and reduce unwanted requests without manually maintaining individual user agent lists. - [Agent Identification API](https://knownagents.com/products/agent-identification-api): Identify agents, verify supported identities, and detect automation so your code can enforce access policies, support content licensing, and enrich logs without maintaining its own agent database. ## Setup Guides - [Send Visit Events](https://knownagents.com/docs/analytics): Connect your website, CDN, or backend to populate Agent Analytics and AI Chat Referral Tracking with bot activity and human referral data. - [Set Up Automatic Robots.txt](https://knownagents.com/docs/robots-txt): Publish automatically updated rules for the bot categories you want to restrict while retaining your existing robots.txt directives. - [Block Bad Bots](https://knownagents.com/docs/blocking): Implement request blocking on WordPress, WordPress VIP, or Cloudflare to restrict unwanted bots, enforce robots.txt rules, or block unidentified automation where supported. - [Use the Agent Identification API](https://knownagents.com/docs/identification): Add agent identity and verification to your backend so it can make access decisions, recognize legitimate automated clients, and enrich request logs. ## REST API Reference - [Analytics REST API](https://knownagents.com/docs/analytics/rest-api): Send single or batched visit events to power bot analytics and AI referral reports; documents request metadata, response timing, authentication, and integration testing. - [Automatic Robots.txt REST API](https://knownagents.com/docs/robots-txt/rest-api): Generate current robots.txt rules for selected agent categories and paths, with exceptions for individual agents or operators, for periodic publication on your website. - [Agent Identification REST API](https://knownagents.com/docs/identification/rest-api): Identify individual or batched requests and retrieve agent identity, verification status, robots.txt policy results, and available automation signals for your own access decisions. ## Platform Integrations - [Akamai Analytics](https://knownagents.com/docs/analytics/akamai): Deliver CDN request logs through Akamai DataStream for agent analytics and AI referral attribution, including user agent, referrer, and response details. - [AWS Analytics](https://knownagents.com/docs/analytics/aws): Analyze bot visits and AI referrals from CloudFront access logs delivered through Amazon Data Firehose. - [AWS Automatic Robots.txt](https://knownagents.com/docs/robots-txt/aws): Use Lambda@Edge with CloudFront to publish current robots.txt rules for selected bot categories and paths. - [BigCommerce Analytics (via Cloudflare)](https://knownagents.com/docs/analytics/cloudflare-worker#before-you-start): Track bot visits and AI referrals to a BigCommerce storefront using a Cloudflare Worker; includes the required Cloudflare setup for the hosted platform. - [Cloudflare Automatic Robots.txt](https://knownagents.com/docs/robots-txt/cloudflare): Add current rules for selected bot categories to your existing robots.txt at Cloudflare's edge using a Worker. - [Cloudflare Logpush Analytics](https://knownagents.com/docs/analytics/cloudflare-logpush): Send Cloudflare Enterprise HTTP request logs, including referrer and signature headers, to power traffic analytics and supported agent verification. - [Cloudflare Worker Analytics](https://knownagents.com/docs/analytics/cloudflare-worker): Capture request and response metadata at Cloudflare's edge for bot analytics and AI referrals, with batching guidance for websites with high traffic volumes. - [Cloudflare Worker Bad Bot Blocking](https://knownagents.com/docs/blocking/cloudflare-worker): Enforce robots.txt disallow rules and reject unidentified requests that meet a configurable automation score threshold using the Agent Identification API; blocked requests receive HTTP 403. - [Fastly Analytics](https://knownagents.com/docs/analytics/fastly): Configure Fastly's HTTPS log streaming to send CDN request data to Known Agents for bot analytics and AI referral tracking. - [Fastly Automatic Robots.txt](https://knownagents.com/docs/robots-txt/fastly): Deploy a Fastly Compute service that adds current bot rules to robots.txt and forwards other traffic to your existing CDN service. - [Google Cloud Analytics](https://knownagents.com/docs/analytics/google-cloud): Route Google Cloud load balancer logs through Cloud Logging and Pub/Sub to analyze bot traffic and AI referrals using authenticated log delivery. - [HubSpot Analytics (via Cloudflare)](https://knownagents.com/docs/analytics/cloudflare-worker#before-you-start): Measure bot activity and AI referrals on HubSpot websites using a Cloudflare Worker; includes the required Cloudflare setup for the hosted platform. - [Node.js Analytics](https://knownagents.com/docs/analytics/nodejs): Use the Node.js SDK to record pageviews from backend requests and responses, with batched event submission for websites with high traffic volumes. - [Node.js Agent Identification](https://knownagents.com/docs/identification/nodejs): Use SDK methods to identify and verify single or batched requests, then apply the results in backend access policies or log enrichment. - [Node.js Automatic Robots.txt](https://knownagents.com/docs/robots-txt/nodejs): Generate rules for selected bot categories with the Node.js SDK, cache them periodically, and serve them from your website's `/robots.txt` endpoint. - [Salesforce Commerce Cloud Analytics (via Cloudflare)](https://knownagents.com/docs/analytics/cloudflare-worker#before-you-start): Measure bot visits and AI referrals to a Salesforce Commerce Cloud storefront through a Cloudflare Worker; includes the required Cloudflare setup. - [Shopify Analytics (via Cloudflare)](https://knownagents.com/docs/analytics/cloudflare-worker#before-you-start): Measure bot visits and AI referrals to a Shopify storefront using a Cloudflare Worker; includes the required Cloudflare setup for the hosted platform. - [Vercel Analytics](https://knownagents.com/docs/analytics/vercel): Send production logs from static files, functions, redirects, and the firewall through a Vercel Log Drain for bot and referral analytics; requires Pro or Enterprise. - [Webflow Analytics (via Cloudflare)](https://knownagents.com/docs/analytics/cloudflare-worker#before-you-start): Track bot visits and human traffic from AI platforms on a Webflow website using a Cloudflare Worker; includes the required Cloudflare setup. - [WordPress Analytics](https://knownagents.com/docs/analytics/wordpress): Enable Agent Analytics and AI Chat Referral Tracking in the WordPress plugin to measure bot activity and human traffic from AI platforms in one project. - [WordPress Automatic Robots.txt](https://knownagents.com/docs/robots-txt/wordpress): Maintain robots.txt rules for selected agent categories through the WordPress plugin, including on WordPress VIP, as new bots are added to the directory. - [WordPress Bad Bot Blocking](https://knownagents.com/docs/blocking/wordpress): Enable the plugin's robots.txt enforcement to return HTTP 403 to disallowed agents and prevent access to your WordPress content. - [WordPress VIP Analytics](https://knownagents.com/docs/analytics/wordpress-vip): Import WordPress VIP edge logs from existing Amazon S3 storage or managed Google Cloud storage to measure bot visits and AI referrals. - [WordPress VIP Bad Bot Blocking](https://knownagents.com/docs/blocking/wordpress-vip): Maintain WordPress VIP edge restrictions for user agent tokens in selected bot categories, with a scheduled GitHub Actions example that refreshes the rules. ## Optional - [Blog](https://knownagents.com/posts): Case studies, product announcements, and tutorials on managing bot traffic and connecting remote tools to AI assistants. - [Dark Visitors Is Now Known Agents](https://knownagents.com/posts/dark-visitors-is-now-known-agents): Why the company changed its name, its approach to beneficial agent traffic, and what the change means for existing integrations. - [Request a Demo](https://knownagents.com/request-a-demo): Request a walkthrough for your website and discuss integration needs; self-serve setup is also available for Enterprise customers. - [Privacy Policy](https://knownagents.com/privacy-policy): How Known Agents collects, uses, shares, retains, and protects personal data, including cookies, analytics metadata, and requests to exercise privacy rights. - [Terms of Service](https://knownagents.com/terms-of-service): Service and subscription terms covering acceptable use, data ownership, API and export restrictions, third-party integrations, and liability. - [Data Processing Agreement](https://knownagents.com/dpa): Terms for processing customer logs and analytics data, including controller and processor roles, security, subprocessors, international transfers, and data deletion. - [Sitemap](https://knownagents.com/sitemap.xml): Machine-readable index of public pages, including individual agent and operator profiles, for discovering directory entries beyond this overview.