Agent Directory

AI Search Crawlers

• AddSearchBot
  AddSearchBot is a web crawler that indexes website content for AddSearch's AI-powered site search solution, collecting data to provide fast and accurate search results.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• AIWebIndex
  AIWebIndex is Lyrenth's scheduled crawler for building an AI-readable index of the public web. Pages are stored and served to AI agents as clean, attributed documents.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• AIWebIndex-Agent
  AIWebIndex-Agent fetches a page into Lyrenth's index when an agent requests a URL that is not yet indexed or explicitly requests a fresh copy.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• amazon-kendra
  amazon-kendra is an Amazon web crawler that indexes authorized web pages for Amazon Kendra search applications.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• Amzn-SearchBot
  Amzn-SearchBot is an AI search crawler operated by Amazon that indexes web content for use in improving Alexa and other Amazon services.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• AndiAgent
  AndiAgent is the web crawler for Andi and indexes public web content for its AI search engine.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• Anomura
  Anomura is Direqt's web crawler that discovers and indexes links and metadata from websites for inclusion in Direqt's AI search results.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• Applebot
  Applebot is Apple's web crawler that gathers data to power search features across Apple's ecosystem including Spotlight, Siri, and Safari search functionality.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• atlassian-bot
  atlassian-bot is Atlassian's AI web crawler that indexes custom websites for Rovo, allowing the indexed content to appear in Rovo Search results and be used by Rovo Chat …
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• AzureAI-SearchBot
  AzureAI-SearchBot is an AI search crawler operated by Microsoft that indexes web content to support Azure AI services and improve search capabilities across Microsoft's A…
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• Channel3Bot
  Channel3Bot is Channel3's web crawler that visits publicly accessible product detail pages to index product information into a universal product catalog. The bot collects…
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• Claude-SearchBot
  Claude-SearchBot is used to create an index of websites that can be surfaced as results in Anthropic's Claude AI assistant search feature.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• Cloudflare-AI-Search
  Cloudflare-AI-Search is Cloudflare's web crawler that retrieves and indexes website content for AI Search, its managed search and retrieval service.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• Cloudflare-AI-Search-External
  Cloudflare-AI-Search-External is a crawler operated by Cloudflare that retrieves website content for AI Search, its managed search and retrieval service.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• Dola-SearchBot
  Dola-SearchBot is a web crawler operated by ByteDance that appears to collect web content for search features in its Dola AI assistant.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• ExaSearchBot
  ExaSearchBot is Exa's crawler for discovering and indexing public web pages so users can find, retrieve, and cite their content through Exa.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• Google-CloudVertexBot
  Google-CloudVertexBot is a web crawler operated by Google that collects content for Google Cloud's Vertex AI Search service. This crawler indexes web pages to power enter…
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• Kimi-SearchBot
  Kimi-SearchBot is a web crawler operated by Moonshot AI that powers Kimi's search functionality. It analyzes and indexes web pages so they can appear in Kimi search resul…
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• LinkupBot
  LinkupBot is a web crawler operated by Linkup that indexes enterprise web content for its AI search platform. The bot visits websites to collect relevant business informa…
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• meta-webindexer
  meta-webindexer is Meta's web crawler that browses the internet to improve search results for Meta AI users. It analyzes online content to make Meta AI's responses more r…
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• MistralAI-Index
  MistralAI-Index is one of Mistral AI's automated web crawlers. It indexes web content to power search capabilities in Vibe, Mistral's AI assistant product.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• OAI-SearchBot
  OAI-SearchBot is OpenAI's web crawler that indexes websites for SearchGPT, collecting and analyzing web content to power AI-driven search results and real-time informatio…
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• PerplexityBot
  PerplexityBot is the web crawler for Perplexity AI that indexes web content to power the AI search engine's real-time information retrieval and answer generation capabili…
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• PetalBot
  PetalBot is Huawei's web crawler that indexes PC and mobile websites to build search databases for Petal Search engine and provide AI-powered content recommendations for …
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• TalarionSearchBot
  TalarionSearchBot is a web crawler operated by Talarion that indexes recent web content for its AI search API and knowledge base.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• xAI-SearchBot
  xAI-SearchBot is a web crawler operated by xAI, the company behind Grok, to collect information from across the web for use in their search and chatbot products. This bot…
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• YandexAdditional
  YandexAdditional is a web crawler operated by Yandex that retrieves website content for YandexGPT quick answers in Yandex Search.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• YandexAdditionalBot
  YandexAdditionalBot is a web crawler operated by Yandex that retrieves website content for YandexGPT quick answers in Yandex Search.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• YepBot
  YepBot is a web crawler operated by Ahrefs that supports Yep, a search engine providing web search results for human users and AI agents.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →
• ZanistaBot
  ZanistaBot is a web crawler operated by Zanista that indexes public web content for its search platform.
  AI Search Crawler

  AI Search Crawler

  Indexes website content to possibly include as citations in AI-powered search results

  See More →