What is HenkBot?

Henkbot crawls the web on behalf of Valyu, an AI search infrastructure provider that indexes content for use in AI-powered retrieval pipelines. You can use Agent Analytics to see when it visits your website.

Agent Type

Expected Behavior

AI data providers are API services that crawl, scrape, and index the web to supply structured data to AI models, agents, and applications. They act as intermediaries between the open web and AI systems, converting web content into LLM-ready formats for training, retrieval-augmented generation (RAG), search, and other AI workflows. Traffic from these services can be high-volume and systematic, as they maintain their own indexes or crawl on-demand in response to API requests from their customers. A single provider may serve thousands of downstream AI applications, amplifying the reach of each crawl.

Details

HenkBot's User Agent String

Access other known user agent strings and recent IP addresses using the API.

How To Block HenkBot With a Robots.txt Rule

In this example, all pages are blocked. You can customize which pages are off-limits by swapping out / for a different disallowed path.

User-agent: HenkBot # https://knownagents.com/agents/henkbot
Disallow: /

Frequently Asked Questions About HenkBot

Should I Block HenkBot?

Consider your priorities. HenkBot crawls websites on behalf of its customers to supply data for AI training, search, and retrieval-augmented generation. Your content may be redistributed to many downstream AI applications through a single provider. You may want to block it if you're concerned about how your content is being used across those systems, or allow it if you value the discoverability and reach it can provide.

How Do I Block HenkBot?

If you want to, you can block or limit HenkBot's access by configuring user agent token rules in your robots.txt file. The best way to do this is using Automatic Robots.txt, which updates automatically as new agents are discovered. While the vast majority of agents operated by reputable companies honor these robots.txt directives, bad actors may choose to ignore them entirely. In that case, you'll need to implement alternative blocking methods such as firewall rules or server-level restrictions. You can verify whether HenkBot is respecting your rules by setting up Agent Analytics to monitor its visits to your website.

Will Blocking HenkBot Hurt My SEO?

Blocking AI data providers has no direct impact on traditional SEO rankings since they don't control search engine indexing. However, these services feed content into AI search engines, RAG pipelines, and conversational AI platforms. Blocking them could reduce your content's representation across multiple AI-powered discovery channels simultaneously, since a single provider may supply data to many downstream applications.

Does HenkBot Access Private Content?

AI data providers typically crawl publicly accessible web content to build their indexes and fulfill API requests. Some providers operate large-scale proxy networks and may attempt to access content aggressively or bypass rate limits. The scope depends on what their customers request and the provider's own indexing priorities. Most focus on public content, but their scale and the diversity of downstream use cases mean your content could be accessed more broadly than with a single-purpose crawler.

How Can I Tell if HenkBot Is Visiting My Website?

Setting up Agent Analytics will give you realtime visibility into HenkBot visiting your website, along with hundreds of other AI agents, crawlers, and scrapers. This will also let you measure human traffic to your website coming from AI search and chat LLM platforms like ChatGPT, Perplexity, and Gemini.

Why Is HenkBot Visiting My Website?

HenkBot crawled your site to fulfill data requests from its customers or to build and maintain its own web index. Your site was likely identified as containing content relevant to AI training datasets, search indexes, or retrieval-augmented generation pipelines. The crawl may have been triggered by a specific customer API request or as part of the provider's broader web indexing efforts.

How Can I Authenticate Visits From HenkBot?

Agent Analytics authenticates agent visits from many agents, letting you know whether each one was actually from that agent, or spoofed by a bad actor. This helps you identify suspicious traffic patterns and make informed decisions about blocking or allowing specific user agents.