Agent Directory
The definitive guide to every known AI agent, crawler, scraper, and other bot on the internet, updated daily.
AI Search Crawlers
-
LinkupBotLinkupBot is a web crawler operated by Linkup that indexes enterprise web content for its AI search platform. The bot visits websites to collect relevant business informa…AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
meta-webindexermeta-webindexer is Meta's web crawler that browses the internet to improve search results for Meta AI users. It analyzes online content to make Meta AI's responses more r…AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
MistralAI-IndexMistralAI-Index is one of Mistral AI's automated web crawlers. It indexes web content to power search capabilities in Vibe, Mistral's AI assistant product.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
OAI-SearchBotOAI-SearchBot is OpenAI's web crawler that indexes websites for SearchGPT, collecting and analyzing web content to power AI-driven search results and real-time informatio…AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
PerplexityBotPerplexityBot is the web crawler for Perplexity AI that indexes web content to power the AI search engine's real-time information retrieval and answer generation capabili…AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
PetalBotPetalBot is Huawei's web crawler that indexes PC and mobile websites to build search databases for Petal Search engine and provide AI-powered content recommendations for …AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
TalarionSearchBotTalarionSearchBot is a web crawler operated by Talarion that indexes recent web content for its AI search API and knowledge base.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
xAI-SearchBotxAI-SearchBot is a web crawler operated by xAI, the company behind Grok, to collect information from across the web for use in their search and chatbot products. This bot…AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
YandexAdditionalYandexAdditional is a web crawler operated by Yandex that retrieves website content for YandexGPT quick answers in Yandex Search.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
YandexAdditionalBotYandexAdditionalBot is a web crawler operated by Yandex that retrieves website content for YandexGPT quick answers in Yandex Search.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
YepBotYepBot is a web crawler operated by Ahrefs that supports Yep, a search engine providing web search results for human users and AI agents.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
ZanistaBotZanistaBot is a web crawler operated by Zanista that indexes public web content for its search platform.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
Archivers
-
Archive-ItArchive-It is a web archiving crawler operated by Internet Archive that preserves copies of web pages for long-term digital preservation and historical record-keeping. It…ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
archive.org_botarchive.org_bot is the Internet Archive's web crawler for the Wayback Machine, systematically crawling and preserving publicly accessible web pages for historical record …ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
Arquivo-web-crawlerArquivo-web-crawler is the Portuguese web archive's bot that systematically crawls and preserves Portuguese websites for historical research, creating a comprehensive dig…ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
AuthoryAuthory is an automated content archiving crawler that systematically searches for and backs up published articles, podcasts, and videos by journalists and content creato…ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
bl.uk_lddc_botbl.uk_lddc_bot is operated by the British Library as part of their legal deposit web archiving program, which collects and preserves UK web content for the national archi…ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
bne.es_botbne.es_bot is an archiver.ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
bnf.fr_botbnf.fr_bot is the official web crawler of the Bibliothèque nationale de France (BNF), systematically collecting and archiving digital content from French websites to pres…ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
CUCH.org Archive BotCUCH.org Archive Bot is a web crawler that archives web content on behalf of C U Cyber History.ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
heritrixheritrix is an archiver operated by Internet Archive.ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
ia_archiveria_archiver is an archiver operated by Internet Archive.ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
ia_archiver-web.archive.orgia_archiver-web.archive.org is an archiver operated by Internet Archive.ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
IABotIABot (InternetArchiveBot) is a Wikipedia bot operated by the Internet Archive that combats link rot by finding dead links on Wikipedia articles and adding archived versi…ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
Internet ArchiveInternet Archive is an archiver operated by Internet Archive.ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
mirrorwebmirrorweb is a web archiving crawler that captures websites in their entirety, including dynamic content and interactive elements, creating tamper-proof WARC archives for…ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
netarkivindsamlingnetarkivindsamling is a web crawler operated by the Royal Danish Library that collects Danish internet content according to the Danish Legal Deposit Act for preservation …ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
NicecrawlerNicecrawler is an archiver operated by NiceCrawler.ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
OddlyBotOddlyBot crawls public web pages to create a verifiable record of what they published and when.ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
SmarshBotSmarshBot is an archiving bot operated by Smarsh that crawls and captures web content for compliance management and regulatory recordkeeping purposes. Organizations use t…ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
special_archiverspecial_archiver is a web crawler operated by Internet Archive as part of their Archive-It service, which creates and preserves collections of web content. This bot visit…ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
TurnitinTurnitin crawler gathers web content to build a comprehensive database for plagiarism detection services, comparing student papers against internet content for academic i…ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
XY-Archive-ComplianceXY-Archive-Compliance creates website archives to help customers meet regulatory and legal compliance requirements for data retention and historical website preservation.ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
-
XY-Archive-Compliance-ArchiverXY-Archive-Compliance-Archiver is an archiving bot operated by XY Planning Network, a membership organization for fee-only financial advisors. This bot archives web conte…ArchiverArchiverCaptures and stores historical website snapshots for long-term digital preservationSee More →
Automated Agents
-
Amazon Bedrock AgentCore BrowserAmazon Bedrock AgentCore Browser is a cloud-based browser service that runs on AWS infrastructure. It enables AI agents and applications to interact with websites, naviga…Automated AgentAutomated AgentAutomates browser interactions programmatically without direct human supervisionSee More →
-
Anchor BrowserAnchor Browser is Anchor's cloud-hosted browser designed to enable AI agents to interact with websites in a human-like manner while maintaining verified, legitimate autom…Automated AgentAutomated AgentAutomates browser interactions programmatically without direct human supervisionSee More →
-
BrowserbaseBrowserbase is a cloud browser platform that enables AI agents and applications to autonomously read, write, and perform tasks on the web. It provides scalable, fast, and…Automated AgentAutomated AgentAutomates browser interactions programmatically without direct human supervisionSee More →
-
Cloudflare Browser RenderingCloudflare Browser Rendering is a serverless browser automation service that runs on Cloudflare Workers. It enables AI agents and applications to control a headless brows…Automated AgentAutomated AgentAutomates browser interactions programmatically without direct human supervisionSee More →
-
HeadlessChromeHeadless Chrome is a tool provided by Google's open source Chromium project that gives software the ability to render web pages in full detail. One of the most common use…Automated AgentAutomated AgentAutomates browser interactions programmatically without direct human supervisionSee More →
-
KernelKernel is a cloud browser service that visits websites for applications and AI agents using its browser automation platform.Automated AgentAutomated AgentAutomates browser interactions programmatically without direct human supervisionSee More →
-
Known Agents BrowserKnown Agents Browser is an automated agent operated by Known Agents.Automated AgentAutomated AgentAutomates browser interactions programmatically without direct human supervisionSee More →
Developer Helpers
-
2ip bot2ip bot is a developer helper operated by 2IP.Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
360monitoring360monitoring is a developer helper operated by 360Monitoring.Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
AccessStatusAccessStatus is a monitoring bot that checks HTTP status codes of web pages to determine if URLs are active, redirected, or returning errors for website uptime monitoring…Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
Acquia optimize (Monsido)Acquia optimize (Monsido) is a developer helper.Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
ActiveComply BotActiveComply Bot is a developer helper operated by ActiveComply.Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
Adagio BotAdagio Bot is a developer helper operated by Adagio.Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
AgentReadinessScannerAgentReadinessScanner is a bot operated by Cloudflare that scans websites to assess their readiness for AI agents by checking for emerging standards such as llms.txt, MCP…Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
AlertSiteAlertSite is SmartBear's synthetic monitoring service that uses automated probes to monitor website availability, performance, and functionality from global monitoring lo…Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
Alertsite by SmartbearAlertsite by Smartbear is a developer helper operated by SmartBear.Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →