Agent Directory
The definitive guide to every known AI agent, crawler, scraper, and other bot on the internet, updated daily.
AI Data Scrapers
-
ERNIEBotERNIEBot is a web crawler operated by Baidu that downloads website content to support ERNIE training and generated answers across Baidu services.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
FacebookBotFacebookBot is a web crawler used by Meta to download training data for its AI speech recognition technology.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
Google-ExtendedGoogle-Extended is a web crawler used by Google to download AI training content for its AI products like the Gemini assistant and its Vertex AI generative APIs.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
GoogleOtherGoogleOther is Google's generic crawler used by various product teams for fetching publicly accessible content, including one-off crawls for internal research and develop…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
GPTBotGPTBot is OpenAI's web crawler that collects public web content used to develop and improve its generative AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
HunyuanHunyuan crawls public web content for Tencent's Hunyuan foundation models and AI applications.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ICC-CrawlerICC-Crawler is NICT's research crawler that automatically collects web pages from the Internet for academic research at Japan's National Institute of Information and Comm…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ImageBotImageBot is a research crawler that collects images for machine learning datasets using img2dataset, an open-source tool for downloading, resizing, and packaging images f…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
imageSpiderimageSpider is a web crawler operated by ByteDance, the company behind TikTok, Douyin, and other content platforms. The bot collects images from websites across the inter…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
Kangaroo BotKangaroo Bot is used by the company Kangaroo LLM to download data to train open source AI models tailored to Australian language and culture.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
KimiBotKimiBot is a web crawler operated by Moonshot AI that collects public web content used to train Kimi models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
KimiCrawlerKimiCrawler is a web crawler operated by Moonshot AI that downloads website content for Kimi and related Moonshot AI services.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
laion-huggingface-processorLAION-huggingface-processor is a web crawler operated by LAION (Large-scale Artificial Intelligence Open Network), a non-profit organization that creates open datasets fo…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
LCCLCC is a web crawler operated by the University of Leipzig that collects text data from websites to build large-scale linguistic corpora for research purposes. The bot ga…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
meta-externalagentmeta-externalagent crawls web content for training AI models and improving Meta's products by indexing content directly across the internet.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
micro-crawlmicro-crawl is an AI data scraper.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
MistralBotMistralBot is a web crawler operated by Mistral, a company that develops frontier language models and enterprise AI platforms. The bot collects data from web pages to sup…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
MoonshotBotMoonshotBot is a web crawler operated by Moonshot AI that downloads website content for Kimi and related Moonshot AI services.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
MoonSpiderMoonSpider is a web crawler operated by Moonshot AI that downloads website content for AI model training and research.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
netEstate Imprint CrawlernetEstate Imprint Crawler is an AI data scraper operated by netEstate.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
omgiliomgili is a web crawler used by Webz.io to maintain a repository of web crawl data that it sells to other companies, including those using it to train AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
PanguBotPanguBot is a web crawler operated by the Chinese company Huawei. It's used to download training data for its multimodal LLM (Large Language Model) called PanGu.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
QwenBotQwenBot is a web crawler operated by Alibaba that downloads website content to support Qwen training and generated answers across Alibaba services.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
recraft-image-datasetrecraft-image-dataset collects publicly accessible images for datasets used to train and improve Recraft's image-generation models and design platform.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ReflectionBotReflectionbot is a web crawler operated by Reflection that collects website content to train its open AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
SalamandraVLMSalamandraVLM is a web crawler operated by Aithlas that downloads website content for AI vision-language model research.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
SBIntuitionsBotSBIntuitionsBot is a web crawler operated by SB Intuitions, a Japanese company that develops generative language models optimized for the Japanese language and culture. T…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
SpiderSpider is a web crawler designed for AI projects, including AI agents, LLMs, RAG systems, and data analysis. It collects and converts web data into multiple formats inclu…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
TimpibotTimpibot is used by Timpi's decentralized network of independent node operators. The index they build can be used to train LLMs (Large Language Models).AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
VelenPublicWebCrawlerVelenPublicWebCrawler is a web crawler developed by Velen for Hunter that analyzes millions of publicly accessible internet pages every month. The bot builds business dat…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
webzio-extendedwebzio-extended is a web crawler used by Webz.io to maintain a repository of web crawl data that it sells to other companies, including those using it to train AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
YiBotYiBot crawls public web content for 01.AI, the company behind the Yi family of foundation models and enterprise AI products.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
AI Search Crawlers
-
AddSearchBotAddSearchBot is a web crawler that indexes website content for AddSearch's AI-powered site search solution, collecting data to provide fast and accurate search results.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
AIWebIndexAIWebIndex is Lyrenth's scheduled crawler for building an AI-readable index of the public web. Pages are stored and served to AI agents as clean, attributed documents.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
AIWebIndex-AgentAIWebIndex-Agent fetches a page into Lyrenth's index when an agent requests a URL that is not yet indexed or explicitly requests a fresh copy.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
amazon-kendraamazon-kendra is an Amazon web crawler that indexes authorized web pages for Amazon Kendra search applications.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
Amzn-SearchBotAmzn-SearchBot is an AI search crawler operated by Amazon that indexes web content for use in improving Alexa and other Amazon services.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
AndiAgentAndiAgent is the web crawler for Andi and indexes public web content for its AI search engine.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
AnomuraAnomura is Direqt's web crawler that discovers and indexes links and metadata from websites for inclusion in Direqt's AI search results.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
ApplebotApplebot is Apple's web crawler that gathers data to power search features across Apple's ecosystem including Spotlight, Siri, and Safari search functionality.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
atlassian-botatlassian-bot is Atlassian's AI web crawler that indexes custom websites for Rovo, allowing the indexed content to appear in Rovo Search results and be used by Rovo Chat …AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
AzureAI-SearchBotAzureAI-SearchBot is an AI search crawler operated by Microsoft that indexes web content to support Azure AI services and improve search capabilities across Microsoft's A…AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
Channel3BotChannel3Bot is Channel3's web crawler that visits publicly accessible product detail pages to index product information into a universal product catalog. The bot collects…AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
Claude-SearchBotClaude-SearchBot is used to create an index of websites that can be surfaced as results in Anthropic's Claude AI assistant search feature.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
Cloudflare-AI-SearchCloudflare-AI-Search is Cloudflare's web crawler that retrieves and indexes website content for AI Search, its managed search and retrieval service.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
Cloudflare-AI-Search-ExternalCloudflare-AI-Search-External is a crawler operated by Cloudflare that retrieves website content for AI Search, its managed search and retrieval service.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
Dola-SearchBotDola-SearchBot is a web crawler operated by ByteDance that appears to collect web content for search features in its Dola AI assistant.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
ExaSearchBotExaSearchBot is Exa's crawler for discovering and indexing public web pages so users can find, retrieve, and cite their content through Exa.AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
Google-CloudVertexBotGoogle-CloudVertexBot is a web crawler operated by Google that collects content for Google Cloud's Vertex AI Search service. This crawler indexes web pages to power enter…AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
Kimi-SearchBotKimi-SearchBot is a web crawler operated by Moonshot AI that powers Kimi's search functionality. It analyzes and indexes web pages so they can appear in Kimi search resul…AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →