Agent Directory
The definitive guide to every known AI agent, crawler, scraper, and other bot on the internet, updated daily.
Intelligence Gatherers
-
WebTrackrCrawlerWebTrackrCrawler is Affsignal's web crawler that gathers intelligence across the web to support affiliate marketing tracking and analysis.Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
webziowebzio is a web crawler operated by Webz.io that collects data from the open, deep, and dark web including news articles, blogs, forums, reviews, and social media content…Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
WhiteCircleTrustBotWhiteCircleTrustBot collects security and compliance information from public company trust centers. White Circle uses the data in its security and safety monitoring produ…Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
WhyNoIPv6BotWhyNoIPv6Bot crawls high-traffic websites each day to measure IPv6 support across DNS, web, and mail services.Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
wknd-botwknd-bot is operated by Wunderkind, a performance marketing platform that identifies website visitors and tracks behavioral signals. This bot is likely used for server-si…Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
YaDirectFetcherYaDirectFetcher is a web crawler operated by Yandex, the Russian search engine and technology company. It gathers data for Yandex.Direct advertising platform to analyze w…Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
Yahoo Ad monitoringYahoo Ad monitoring crawls webpages where Yahoo advertisements are served to monitor content quality, ad placement compliance, and advertiser policy adherence.Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
Yahoo JapanYahoo Japan is an intelligence gatherer operated by Yahoo.Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
YaKYaK is a web crawler operated by Linkfluence, a company that provides social media intelligence and analytics services. The bot collects web content for market research, …Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
Yext IncYext Inc is an intelligence gatherer operated by Yext.Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
YextBotYextBot is a web crawler operated by Yext that helps customers retrieve and organize content from their websites to maintain consistent brand information across digital c…Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
ZoominfoBotZoominfoBot crawls corporate websites, press releases, SEC filings and online sources using natural language processing to gather business and professional data.Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
ZveloZvelo fetches public web pages so zvelo can categorize websites for its web classification services.Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
Scrapers
-
008008 is a scraper operated by 80legs.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
AdsTxtCrawlerAdsTxtCrawler is a bot operated by OneTag that crawls and retrieves ads.txt files from websites. These files are part of an Interactive Advertising Bureau standard that p…ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
dcrawldcrawl is a scraper.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
HelloworkJobPostingBotHelloworkJobPostingBot is a web crawler that aggregates job listings by scanning company career pages and job posting websites to collect employment opportunities for the…ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
HTTrackHTTrack is a scraper.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
HTTrack 3.0HTTrack 3.0 is a scraper.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
IndeedJobBotIndeedJobBot is Indeed's web scraper that crawls and indexes job postings across the internet to populate the Indeed job search platform with comprehensive employment opp…ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
KrawlerBotKrawlerBot is a web scraping bot operated by Krawler that crawls and extracts data from websites for its customers.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
MetaInspectorMetaInspector is a scraper.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
MetaJobBotMetaJobBot is a web scraper operated by METAJob, a job search platform based in Germany. The bot collects job listings and employment information to populate their job bo…ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
newspapernewspaper is a scraper.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
NutchNutch is an open-source web crawler developed by the Apache Software Foundation. It is commonly used for large-scale web scraping.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
nyt_scrapingnyt_scraping is a web scraping bot operated by The New York Times. It is used to collect content from websites for The New York Times' purposes.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
Offline ExplorerOffline Explorer is a scraper operated by MetaProducts.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
OpenindexSpiderOpenindexSpider is a scraper operated by Openindex.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
PotionsPotions is a web crawler that fetches product feeds and crawls data from e-commerce websites to collect product information for commercial purposes.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
ScrapyScrapy is an open source web scraping framework written in Python that allows developers to extract data from websites. It is maintained by Zyte and used by millions of d…ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
ServerHunterSpiderServerHunterSpider indexes hosting-plan prices, specifications, and availability across providers for ServerHunter's server comparison database.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
StatsDroneBotStatsDroneBot is StatsDrone's web scraper that collects and aggregates affiliate marketing statistics and performance data from various websites and platforms.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
ThinkbotThinkbot is a scraper.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
-
web-image-collection-researchweb-image-collection-research is a web scraper that collects public images for a research dataset.ScraperScraperExtracts large amounts of web data, often without explicit website permissionSee More →
Search Engine Crawlers
-
360Spider360Spider is a search engine crawler operated by Qihoo 360.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Alexa ArchiveAlexa Archive is a search engine crawler operated by Alexandria.org.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
alexa site auditalexa site audit was a web crawler operated by Amazon as part of the Alexa Internet service, which provided website analytics and traffic data. The bot collected informat…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
AlexandriaOrgBotAlexandriaOrgBot is a search engine crawler operated by Alexandria.org.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
AlgoliaAlgolia is a search engine crawler operated by Algolia.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Algolia CrawlerAlgolia Crawler is a web crawler that indexes website content for Algolia's search-as-a-service platform, enabling fast and relevant search functionality for applications…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Atom Feed RobotAtom Feed Robot is a search engine crawler operated by RSSMicro.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
BaiduspiderBaiduspider is Baidu's web crawler that indexes web content for China's largest search engine, crawling and analyzing websites to provide search results for Baidu users.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Baiduspider-renderBaiduspider-render is a web crawler operated by Baidu, China's largest search engine, specifically designed to render and index JavaScript-heavy pages. This variant proce…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
bingbotbingbot is Microsoft's standard web crawler that handles most of Bing's crawling needs each day, indexing web content for Bing search results using both desktop and mobil…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
CGMSearchBotCGMSearchBot indexes submitted websites and sitemaps so their pages can appear in CGMIMM search results.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
cludo.com botcludo.com bot is a crawler operated by Cludo, a site search platform that provides search and discovery solutions for websites. The bot likely crawls and indexes website …Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Cốc CốcCốc Cốc is a search engine crawler.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
coccocbotcoccocbot is the web crawler for Coc Coc, a Vietnamese search engine and browser. It indexes websites to provide search results for Vietnamese users.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
coccocbot-imagecoccocbot-image is the image crawler for Cốc Cốc, a Vietnamese search engine and web browser. This bot crawls and indexes images from websites to populate image search re…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
coccocbot-webcoccocbot-web is the web crawler for Cốc Cốc, a Vietnamese search engine and web browser. This bot indexes content for inclusion in Cốc Cốc's search results, a popular in…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →