Agent Directory
The definitive guide to every known AI agent, crawler, scraper, and other bot on the internet, updated daily.
Intelligence Gatherers
YextBot
YextBot is a web crawler operated by Yext that helps customers retrieve and organize content from their websites to maintain consistent brand information across digital c…
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
ZoominfoBot
ZoominfoBot crawls corporate websites, press releases, SEC filings and online sources using natural language processing to gather business and professional data.
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
Scrapers
008
008 is a scraper operated by 80legs.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
AdsTxtCrawler
AdsTxtCrawler is a bot operated by OneTag that crawls and retrieves ads.txt files from websites. These files are part of an Interactive Advertising Bureau standard that p…
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
dcrawl
dcrawl is a scraper.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
HelloworkJobPostingBot
HelloworkJobPostingBot is a web crawler that aggregates job listings by scanning company career pages and job posting websites to collect employment opportunities for the…
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
HTTrack
HTTrack is a scraper.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
HTTrack 3.0
HTTrack 3.0 is a scraper.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
IndeedJobBot
IndeedJobBot is Indeed's web scraper that crawls and indexes job postings across the internet to populate the Indeed job search platform with comprehensive employment opp…
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
KrawlerBot
KrawlerBot is a web scraping bot operated by Krawler that crawls and extracts data from websites for its customers.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
MetaInspector
MetaInspector is a scraper.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
MetaJobBot
MetaJobBot is a web scraper operated by METAJob, a job search platform based in Germany. The bot collects job listings and employment information to populate their job bo…
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
newspaper
newspaper is a scraper.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
Nutch
Nutch is an open-source web crawler developed by the Apache Software Foundation. It is commonly used for large-scale web scraping.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
nyt_scraping
nyt_scraping is a web scraping bot operated by The New York Times. It is used to collect content from websites for The New York Times' purposes.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
Offline Explorer
Offline Explorer is a scraper operated by MetaProducts.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
OpenindexSpider
OpenindexSpider is a scraper operated by Openindex.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
Potions
Potions is a web crawler that fetches product feeds and crawls data from e-commerce websites to collect product information for commercial purposes.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
Scrapy
Scrapy is an open source web scraping framework written in Python that allows developers to extract data from websites. It is maintained by Zyte and used by millions of d…
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
ServerHunterSpider
ServerHunterSpider is a specialized web crawler that indexes pricing, specifications, and stock information of hosting plans from 850+ providers, fully respecting robots.…
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
StatsDroneBot
StatsDroneBot is StatsDrone's web scraper that collects and aggregates affiliate marketing statistics and performance data from various websites and platforms.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
Thinkbot
Thinkbot is a scraper.
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
Search Engine Crawlers
360Spider
360Spider is a search engine crawler operated by Qihoo 360.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
Alexa Archive
Alexa Archive is a search engine crawler operated by Alexandria.org.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
alexa site audit
alexa site audit was a web crawler operated by Amazon as part of the Alexa Internet service, which provided website analytics and traffic data. The bot collected informat…
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
AlexandriaOrgBot
AlexandriaOrgBot is a search engine crawler operated by Alexandria.org.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
Algolia
Algolia is a search engine crawler operated by Algolia.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
Algolia Crawler
Algolia Crawler is a web crawler that indexes website content for Algolia's search-as-a-service platform, enabling fast and relevant search functionality for applications…
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
Atom Feed Robot
Atom Feed Robot is a search engine crawler operated by RSSMicro.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
Baiduspider
Baiduspider is Baidu's web crawler that indexes web content for China's largest search engine, crawling and analyzing websites to provide search results for Baidu users.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
Baiduspider-render
Baiduspider-render is a web crawler operated by Baidu, China's largest search engine, specifically designed to render and index JavaScript-heavy pages. This variant proce…
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
bingbot
bingbot is Microsoft's standard web crawler that handles most of Bing's crawling needs each day, indexing web content for Bing search results using both desktop and mobil…
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
cludo.com bot
cludo.com bot is a crawler operated by Cludo, a site search platform that provides search and discovery solutions for websites. The bot likely crawls and indexes website …
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
Cốc Cốc
Cốc Cốc is a search engine crawler.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
coccocbot
coccocbot is the web crawler for Coc Coc, a Vietnamese search engine and browser. It indexes websites to provide search results for Vietnamese users.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
coccocbot-image
coccocbot-image is the image crawler for Cốc Cốc, a Vietnamese search engine and web browser. This bot crawls and indexes images from websites to populate image search re…
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
coccocbot-web
coccocbot-web is the web crawler for Cốc Cốc, a Vietnamese search engine and web browser. This bot indexes content for inclusion in Cốc Cốc's search results, a popular in…
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
Coveo Bot
Coveo Bot is a search engine crawler operated by Coveo.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
Coveobot
Coveobot is a web crawler that indexes website content for Coveo's enterprise search platform, following hyperlinks to discover and index pages for customer search interf…
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
crawler.freespoke.com
crawler.freespoke.com is the web crawler for Freespoke, an alternative search engine that indexes web content to provide uncensored search results. The search engine cove…
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
Crawlson
Crawlson is a web crawler bot that indexes content for the Crawlson search engine by systematically browsing and analyzing web pages.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
Daum
Daum is a major Korean search engine crawler operated by Kakao Corp, systematically indexing Korean websites and providing comprehensive search services including web, ne…
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
DocSpotBot
DocSpotBot is a web crawler operated by DocSpot. It's used to crawl web pages to help people find doctors and other medical providers.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
DuckDuckBot
DuckDuckBot is DuckDuckGo's web crawler that constantly improves search results by indexing web content, respecting robots.txt and providing users the best search experie…
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
DuckDuckGo-Favicons-Bot
DuckDuckGo-Favicons-Bot is a crawler operated by DuckDuckGo, the privacy-focused search engine. This bot collects favicon images from websites to display site icons in se…
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
Feedfetcher-Google
Feedfetcher-Google is a crawler operated by Google that fetches RSS and Atom feeds for various Google products and services. The bot retrieves syndication feeds to help i…
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
FindFiles.net
FindFiles.net is a search engine crawler that indexes web content for the FindFiles.net search engine.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
FindITAnswersbot
FindITAnswersbot is a search engine crawler operated by IT Influentials.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
Freespoke
Freespoke is a search engine crawler operated by Freespoke.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →
FreespokeCrawler
FreespokeCrawler is a search engine crawler operated by Freespoke. It visits websites to index and collect content for inclusion in Freespoke's search engine results.
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →