Agent Directory
The definitive guide to every known AI agent, crawler, scraper, and other bot on the internet, updated daily.
AI Coding Agents
-
opencodeOpenCode is an open-source AI coding agent that helps developers write code from the terminal, IDE, or desktop, supporting multiple LLM providers and local models.AI Coding AgentAI Coding AgentFetches documentation and other resources to help build softwareSee More →
-
qodercliqodercli is a command-line AI coding agent operated by Qoder that helps developers build, debug, and modify software from the terminal.AI Coding AgentAI Coding AgentFetches documentation and other resources to help build softwareSee More →
-
TraeTrae is an AI-powered coding agent developed by ByteDance that can understand codebases, fetch web content, and generate code.AI Coding AgentAI Coding AgentFetches documentation and other resources to help build softwareSee More →
AI Data Providers
-
AgentDataBotAgentDataBot visits public web pages to identify technology stacks, company details, and team members. The data builds AgentData's company-intelligence database for AI ag…AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
ApifyBotApifyBot is a web scraping and data extraction crawler by Apify that collects website content for use in AI, LLMs, RAG, and automation workflows.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
ApifyWebsiteContentCrawlerApifyWebsiteContentCrawler is a web crawler by Apify that extracts and downloads full website content for use in AI, data analysis, and automation workflows.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
artsdata-crawlerartsdata-crawler is operated by Culture Creates and collects publicly available arts and culture information for the Artsdata knowledge graph.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
BixelBotBixelBot collects public company and market signals for Bixel's structured company-data platform, which supplies verified business information to AI agents and developers…AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
BravebotBravebot is a web crawler by Brave that indexes pages for Brave Search, providing search data and AI-optimized context to power chatbots, agents, and RAG pipelines.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
BrightbotBrightbot is a web data collection crawler by Bright Data that extracts and structures public website content at scale, providing AI-ready data for model training, RAG pi…AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
CloudflareBrowserRenderingCrawlerCloudflareBrowserRenderingCrawler is a web crawler operated by Cloudflare that returns rendered website content for research, monitoring, and AI data workflows.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
CragCrawlerCragCrawler is a web scraping bot operated by CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services. The bot is used…AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
CrawlspaceCrawlspace is a web crawler platform that fetches and extracts website content for AI agents, RAG applications, and structured data workflows.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
DesearchBotDesearchBot crawls and indexes public web content for Desearch's search engine and search APIs.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
DiffbotDiffbot is a web crawler that extracts and structures website content using AI-powered visual understanding, providing knowledge graph data for applications like market i…AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
ExaBotExaBot is a web crawler that indexes web content to power Exa's AI search engine and semantic search APIs for AI applications.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
FirecrawlAgentFirecrawlAgent is a web crawler operated by Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
HenkBotHenkbot crawls the web on behalf of Valyu, an AI search infrastructure provider that indexes content for use in AI-powered retrieval pipelines.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
Keenable-UserKeenable-User accesses web content for Keenable's search and content-access infrastructure for AI agents and AI labs.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
KeenableBotKeenableBot crawls web pages for Keenable's independent search index. Keenable provides this index through search and content-retrieval APIs used by AI agents, AI labs, a…AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
Mozilla-TabstackMozilla-Tabstack is an AI agent operated by Mozilla that performs programmatic, AI-driven interactions with web content through Tabstack.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
Quartr CrawlerQuartr Crawler collects public company information for Quartr's data products, which make financial material available to researchers and applications.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
Querit-SearchBotQuerit-SearchBot is a web crawler operated by Querit that indexes web content for their search API service, which is designed to provide real-time search results for larg…AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
QueritBotQueritBot is a web crawler operated by Querit, a company providing a search API for large language model integration. This bot indexes web content to power the real-time …AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
ScryBotScryBot collects public web content for Scry, Unflatten's programmable search platform. Scry lets applications query and analyze an index spanning web pages, social conte…AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
ShapBotShapBot is a web crawler by Parallel that collects and structures web content to power its search, extraction, and deep research APIs, providing AI agents with high-accur…AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
SofyaBotSofyaBot crawls public web pages to build the independent search index behind Sofya's API, which gives AI agents access to current web information.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
TalarionFrontierCrawlerTalarionFrontierCrawler is a web crawler operated by Talarion that collects public web content for its research corpus and AI knowledge services.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
TavilyBotTavilyBot is a web crawler by Tavily that indexes and extracts content from billions of pages, providing real-time search, extraction, and research data to ground AI agen…AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
Terra CottaTerra Cotta is Ceramic's web crawler that indexes public content to power their web-scale search API for AI and LLMs.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
TerraCottaTerraCotta is Ceramic's web crawler that indexes public content to power their web-scale search API for AI and LLMs.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
-
YouBotYouBot is You.com's web crawler that indexes and extracts public content for its search, content, and research APIs, supplying web data to AI applications.AI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party serviceSee More →
AI Data Scrapers
-
AI2BotAI2Bot is operated by Ai2, a non-profit AI research institute. It's used to download data to train open source AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
Ai2Bot-DolmaAi2Bot-Dolma is operated by Ai2, a non-profit AI research institute. It's used to download data to train open source AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
AmazonbotAmazonbot is a web crawler operated by Amazon that builds an index of web content to improve Amazon products and services, including Alexa, Kindle, and Amazon Shopping. T…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
Applebot-ExtendedApplebot-Extended is an Apple robots.txt control token that lets publishers opt out of using Applebot-crawled content to train Apple foundation models. It does not crawl …AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
BeansLLM-CorpusBotBeansLLM-CorpusBot collects public web content to build corpora for language-model training.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
bedrockbotbedrockbot is a web crawler operated by Amazon that crawls selected URLs for Amazon Bedrock knowledge bases and custom AI applications.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
BytespiderBytespider is a web crawler operated by ByteDance, the Chinese owner of TikTok. It's allegedly used to download training data for its LLMs (Large Language Model) includin…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
CCBotCCBot is Common Crawl's web crawler that creates an open repository of web data, making crawled content universally accessible for research, analysis, and AI training pur…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ChatGLM-SpiderChatGLM-Spider is a web crawler operated by Zhipu AI, the Chinese company behind ChatGLM. It is used for collecting data to train and evaluate the company's large languag…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ClaudeBotClaudeBot is a web crawler operated by Anthropic to download training data for its LLMs (Large Language Models) that power AI products like Claude.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
CloudVertexBotCloudVertexBot is a Google-operated crawler available to site owners to request targeted crawls of their own sites for AI training purposes on the Vertex AI platform.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
cohere-training-data-crawlercohere-training-data-crawler is a web crawler operated by Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
CohereBotCohereBot is a web crawler operated by Cohere, a company that provides large language models and enterprise solutions. This bot collects data from websites to support the…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
CotoyogiCotoyogi is a research crawler operated by Japan's Research Organization of Information and Systems that collects web content to build AI training datasets for research p…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
Datenbank CrawlerDatenbank Crawler is a web crawler operated by German company netEstate used for collecting and selling international website data.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
DeepSeekBotDeepSeekBot is a web crawler operated by DeepSeek that downloads website content for AI model training and other data collection use cases.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
DoubaobotDoubaobot is a web crawler operated by ByteDance that downloads website content to support Doubao training and generated answers.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ElevenlabsCrawlerElevenlabsCrawler is ElevenLabs' web crawler that collects public content that may be used to train its generative AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →