What is Claude-Code?
Claude Code is an AI coding agent by Anthropic that can build, debug, and ship code directly from the terminal, handling tasks like codebase onboarding, multi-file edits, and turning issues into pull requests. You can use Known Agents (formerly Dark Visitors) Agent Analytics to see when Claude-Code visits your website.
Agent Type
Expected Behavior
AI coding agents are autonomous programs that browse websites to gather information needed to complete software development tasks. They fetch documentation, API references, code examples, changelogs, and technical guides to assist with writing, debugging, or refactoring code. Unlike general AI agents, their browsing is focused on technical content and driven by specific coding objectives. Request patterns vary based on the complexity of the development task and may involve fetching multiple pages from documentation sites in quick succession.
Detail
| Operated By | Anthropic |
| Last Updated | 2 hours ago |
Top Website Robots.txts
Country of Origin
Top Website Blocking Trend Over Time
The percentage of the world's top 1000 websites who are blocking Claude-Code
Overall AI Coding Agent Traffic
The percentage of all internet traffic coming from AI coding agents
Robots.txt
In this example, all pages are blocked. You can customize which pages are off-limits by swapping out / for a different disallowed path.
User-agent: Claude-Code # https://knownagents.com/agents/claude-code
Disallow: /Frequently Asked Questions About Claude-Code
Should I Block Claude-Code?
Generally no. Claude-Code fetches documentation and technical content to help human software engineers write and debug code. Blocking it prevents them from efficiently accessing your technical resources. Allowing Claude-Code also helps developers discover your APIs, libraries, and documentation in the first place.
How Do I Block Claude-Code?
If you want to, you can block or limit Claude-Code's access by configuring user agent token rules in your robots.txt file. The best way to do this is using Automatic Robots.txt, which update automatically as new agents are discovered. While the vast majority of agents operated by reputable companies honor these robots.txt directives, bad actors may choose to ignore them entirely. In that case, you'll need to implement alternative blocking methods such as firewall rules or server-level restrictions. You can verify whether Claude-Code is respecting your rules by setting up Agent Analytics to monitor its visits to your website.
Will Blocking Claude-Code Hurt My SEO?
Blocking AI coding agents is unlikely to directly impact traditional SEO rankings since they don't contribute to search engine indexing. However, if your site hosts technical documentation or other developer resources, blocking these agents could reduce developer engagement and adoption of your tools, which may indirectly affect your site's authority over time.
Does Claude-Code Access Private Content?
AI coding agents typically access publicly available technical content such as documentation pages, API references, code repositories, and developer guides. They generally don't attempt to access private or authenticated content unless explicitly configured with credentials by the developer using them. Their scope is usually limited to technical resources relevant to the coding task at hand.
How Can I Tell if Claude-Code Is Visiting My Website?
Setting up Agent Analytics will give you realtime visibility into Claude-Code visiting your website, along with hundreds of other AI agents, crawlers, and scrapers. This will also let you measure human traffic to your website coming from AI search and chat LLM platforms like ChatGPT, Perplexity, and Gemini.
Why Is Claude-Code Visiting My Website?
Claude-Code visited your site because a developer is using an AI coding tool that identified your content as relevant to their development task. Your site likely contains documentation, API references, code examples, or technical guides that the agent fetched to help the developer write, debug, or understand code.
How Can I Authenticate Visits From Claude-Code?
Agent Analytics authenticates agent visits from many agents, letting you know whether each one was actually from that agent, or spoofed by a bad actor. This helps you identify suspicious traffic patterns and make informed decisions about blocking or allowing specific user agents.