Agent Directory
The definitive guide to every known AI agent, crawler, scraper, and other bot on the internet, updated daily.
AI Data Scrapers
-
AI2BotAI2Bot is operated by Ai2, a non-profit AI research institute. It's used to download data to train open source AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
Ai2Bot-DolmaAi2Bot-Dolma is operated by Ai2, a non-profit AI research institute. It's used to download data to train open source AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
AmazonbotAmazonbot is a web crawler operated by Amazon that builds an index of web content to improve Amazon products and services, including Alexa, Kindle, and Amazon Shopping. T…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
Applebot-ExtendedApplebot-Extended is an Apple robots.txt control token that lets publishers opt out of using Applebot-crawled content to train Apple foundation models. It does not crawl …AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
BeansLLM-CorpusBotBeansLLM-CorpusBot collects public web content to build corpora for language-model training.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
bedrockbotbedrockbot is a web crawler operated by Amazon that crawls selected URLs for Amazon Bedrock knowledge bases and custom AI applications.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
BytespiderBytespider is a web crawler operated by ByteDance, the Chinese owner of TikTok. It's allegedly used to download training data for its LLMs (Large Language Model) includin…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
CCBotCCBot is Common Crawl's web crawler that creates an open repository of web data, making crawled content universally accessible for research, analysis, and AI training pur…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ChatGLM-SpiderChatGLM-Spider is a web crawler operated by Zhipu AI, the Chinese company behind ChatGLM. It is used for collecting data to train and evaluate the company's large languag…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ClaudeBotClaudeBot is a web crawler operated by Anthropic to download training data for its LLMs (Large Language Models) that power AI products like Claude.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
CloudVertexBotCloudVertexBot is a Google-operated crawler available to site owners to request targeted crawls of their own sites for AI training purposes on the Vertex AI platform.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
cohere-training-data-crawlercohere-training-data-crawler is a web crawler operated by Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
CohereBotCohereBot is a web crawler operated by Cohere, a company that provides large language models and enterprise solutions. This bot collects data from websites to support the…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
CotoyogiCotoyogi is a research crawler operated by Japan's Research Organization of Information and Systems that collects web content to build AI training datasets for research p…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
Datenbank CrawlerDatenbank Crawler is a web crawler operated by German company netEstate used for collecting and selling international website data.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
DeepSeekBotDeepSeekBot is a web crawler operated by DeepSeek that downloads website content for AI model training and other data collection use cases.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
DoubaobotDoubaobot is a web crawler operated by ByteDance that downloads website content to support Doubao training and generated answers.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ElevenlabsCrawlerElevenlabsCrawler is ElevenLabs' web crawler that collects public content that may be used to train its generative AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ERNIEBotERNIEBot is a web crawler operated by Baidu that downloads website content to support ERNIE training and generated answers across Baidu services.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
FacebookBotFacebookBot is a web crawler used by Meta to download training data for its AI speech recognition technology.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
Google-ExtendedGoogle-Extended is a web crawler used by Google to download AI training content for its AI products like the Gemini assistant and its Vertex AI generative APIs.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
GoogleOtherGoogleOther is Google's generic crawler used by various product teams for fetching publicly accessible content, including one-off crawls for internal research and develop…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
GPTBotGPTBot is OpenAI's web crawler that collects public web content used to develop and improve its generative AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
HunyuanHunyuan crawls public web content for Tencent's Hunyuan foundation models and AI applications.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ICC-CrawlerICC-Crawler is NICT's research crawler that automatically collects web pages from the Internet for academic research at Japan's National Institute of Information and Comm…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ImageBotImageBot is a research crawler that collects images for machine learning datasets using img2dataset, an open-source tool for downloading, resizing, and packaging images f…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
imageSpiderimageSpider is a web crawler operated by ByteDance, the company behind TikTok, Douyin, and other content platforms. The bot collects images from websites across the inter…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
Kangaroo BotKangaroo Bot is used by the company Kangaroo LLM to download data to train open source AI models tailored to Australian language and culture.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
KimiBotKimiBot is a web crawler operated by Moonshot AI that collects public web content used to train Kimi models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
KimiCrawlerKimiCrawler is a web crawler operated by Moonshot AI that downloads website content for Kimi and related Moonshot AI services.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
laion-huggingface-processorLAION-huggingface-processor is a web crawler operated by LAION (Large-scale Artificial Intelligence Open Network), a non-profit organization that creates open datasets fo…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
LCCLCC is a web crawler operated by the University of Leipzig that collects text data from websites to build large-scale linguistic corpora for research purposes. The bot ga…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
meta-externalagentmeta-externalagent crawls web content for training AI models and improving Meta's products by indexing content directly across the internet.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
micro-crawlmicro-crawl is an AI data scraper.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
MistralBotMistralBot is a web crawler operated by Mistral, a company that develops frontier language models and enterprise AI platforms. The bot collects data from web pages to sup…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
MoonshotBotMoonshotBot is a web crawler operated by Moonshot AI that downloads website content for Kimi and related Moonshot AI services.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
MoonSpiderMoonSpider is a web crawler operated by Moonshot AI that downloads website content for AI model training and research.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
netEstate Imprint CrawlernetEstate Imprint Crawler is an AI data scraper operated by netEstate.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
omgiliomgili is a web crawler used by Webz.io to maintain a repository of web crawl data that it sells to other companies, including those using it to train AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
PanguBotPanguBot is a web crawler operated by the Chinese company Huawei. It's used to download training data for its multimodal LLM (Large Language Model) called PanGu.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
QwenBotQwenBot is a web crawler operated by Alibaba that downloads website content to support Qwen training and generated answers across Alibaba services.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
recraft-image-datasetrecraft-image-dataset collects publicly accessible images for datasets used to train and improve Recraft's image-generation models and design platform.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ReflectionBotReflectionbot is a web crawler operated by Reflection that collects website content to train its open AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
SalamandraVLMSalamandraVLM is a web crawler operated by Aithlas that downloads website content for AI vision-language model research.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
SBIntuitionsBotSBIntuitionsBot is a web crawler operated by SB Intuitions, a Japanese company that develops generative language models optimized for the Japanese language and culture. T…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
SpiderSpider is a web crawler designed for AI projects, including AI agents, LLMs, RAG systems, and data analysis. It collects and converts web data into multiple formats inclu…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
TimpibotTimpibot is used by Timpi's decentralized network of independent node operators. The index they build can be used to train LLMs (Large Language Models).AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
VelenPublicWebCrawlerVelenPublicWebCrawler is a web crawler developed by Velen for Hunter that analyzes millions of publicly accessible internet pages every month. The bot builds business dat…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
webzio-extendedwebzio-extended is a web crawler used by Webz.io to maintain a repository of web crawl data that it sells to other companies, including those using it to train AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
YiBotYiBot crawls public web content for 01.AI, the company behind the Yi family of foundation models and enterprise AI products.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
Page 1 of 1