Agent Directory
The definitive guide to every known AI agent, crawler, scraper, and other bot on the internet, updated daily.
Search Engine Crawlers
-
Coveo BotCoveo Bot is a search engine crawler operated by Coveo.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
CoveobotCoveobot is a web crawler that indexes website content for Coveo's enterprise search platform, following hyperlinks to discover and index pages for customer search interf…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
crawler.freespoke.comcrawler.freespoke.com is the web crawler for Freespoke, an alternative search engine that indexes web content to provide uncensored search results. The search engine cove…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
CrawlsonCrawlson is a web crawler bot that indexes content for the Crawlson search engine by systematically browsing and analyzing web pages.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
DaumDaum is a major Korean search engine crawler operated by Kakao Corp, systematically indexing Korean websites and providing comprehensive search services including web, ne…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
DocSpotBotDocSpotBot is a web crawler operated by DocSpot. It's used to crawl web pages to help people find doctors and other medical providers.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
DuckDuckBotDuckDuckBot is DuckDuckGo's web crawler that indexes public web pages to improve the coverage and quality of DuckDuckGo search results.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
DuckDuckGo-Favicons-BotDuckDuckGo-Favicons-Bot is a crawler operated by DuckDuckGo, the privacy-focused search engine. This bot collects favicon images from websites to display site icons in se…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
EtherdeckBotEtherdeckBot crawls and indexes public web pages for Etherdeck's search engine.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Feedfetcher-GoogleFeedfetcher-Google is a crawler operated by Google that fetches RSS and Atom feeds for various Google products and services. The bot retrieves syndication feeds to help i…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
FindFiles.netFindFiles.net is a search engine crawler that indexes web content for the FindFiles.net search engine.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
FindiseBotFindiseBot crawls and indexes public web pages for Findise, an independent search engine focused on private, unbiased web search.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
FindITAnswersbotFindITAnswersbot is a search engine crawler operated by IT Influentials.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
FreespokeFreespoke is a search engine crawler operated by Freespoke.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
FreespokeCrawlerFreespokeCrawler is a search engine crawler operated by Freespoke. It visits websites to index and collect content for inclusion in Freespoke's search engine results.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
FrogleBotFrogleBot is a search engine crawler operated by Frogle that crawls and indexes public pages for its search engine.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
FunnelbackFunnelback is a search engine crawler operated by Squiz, a digital experience platform company. It indexes web content to power enterprise search functionality, leveragin…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
fyndbotfyndbot is the web crawler for Fynd, an independent search engine. This bot visits websites to discover and update its search index with fresh content.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
FyndSearchEngine-CrawlerFyndSearchEngine-Crawler is a web crawler associated with fynd, a search engine service. This bot visits websites to index content for inclusion in the fynd search engine…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
FyndSearchEngine-ReCrawlerFyndSearchEngine-ReCrawler is a web crawler operated by fynd, a search engine service. This bot specifically handles re-crawling operations to update fynd's search index …Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
GeedoProductSearchGeedoProductSearch is a crawler operated by Geedo, a product search engine that indexes billions of products from millions of online stores worldwide. The bot visits e-co…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
GidoraBotGidoraBot is the web crawler for the Gidora search engine and indexes public pages for its search results.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
GigabotGigabot is the web crawler used by Gigablast, an independent U.S. search engine. The crawler indexes web content using proprietary search algorithms and database systems.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Google FaviconGoogle Favicon is a search engine crawler operated by Google.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Google ImagesGoogle Images is a search engine crawler operated by Google.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Google ScholarGoogle Scholar is a search engine crawler operated by Google.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Google VideosGoogle Videos is a search engine crawler operated by Google.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
GooglebotGooglebot is Google's web crawler that discovers and indexes web content for Google Search, including both mobile and desktop variants that crawl websites to understand t…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Googlebot-IAGooglebot-IA is Google Scholar's specialized crawler that indexes scholarly literature from academic publishers, repositories, and university websites to populate the aca…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Googlebot-ImageGooglebot-Image is Google's specialized search engine crawler that discovers and indexes images across the web for inclusion in Google Images search results, analyzing vi…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Googlebot-MobileGooglebot-Mobile is the mobile crawling component of Google's web crawler system used to index and analyze mobile versions of websites for inclusion in Google's mobile se…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Googlebot-NewsGooglebot-News is a search engine crawler operated by Google.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Googlebot-VideoGooglebot-Video is Google's specialized crawler for video content discovery and indexing. It crawls websites to find and analyze video files, metadata, and thumbnails for…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Greppr Web CrawlerGreppr Web Crawler is a search engine crawler that indexes web content for Greppr, a search engine that focuses on providing unfiltered search results without tracking or…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
HaosouSpiderHaosouSpider is a search engine crawler operated by Qihoo 360.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Hype MachineHype Machine crawls music publications and blogs since 2005 to discover new artist posts and builds curated playlists from this metadata for music listeners and discovery…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
ibou-discoveryibou-discovery crawls public articles for Ibou Discovery, which recommends stories from a range of sources and links readers to the original publishers.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
ibou-imagesibou-images is Ibou's crawler for images on public websites, gathering visual content for its search products.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
ibou-newsibou-news crawls public news articles for Ibou News, which groups coverage by topic and links readers to the original publishers.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
ibou-searchibou-search is Ibou's main web crawler. It discovers and indexes public pages for Ibou's search engine, which cites sources and links readers to the original publishers.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
IbouBotIbouBot is Ibou's development and testing crawler. Ibou uses separate user agents, including ibou-search, for production search crawling and indexing.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
intelx.io_botintelx.io_bot is the web crawler for Intelligence X, a search engine that indexes and archives internet data including emails, domains, IPs, URLs, and other identifiable …Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
IsonSearchBotIsonSearchBot crawls public web pages to build the Ison Search index, prioritizing Indonesian sources.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
JambotJambot is a web crawler operated by JamBot company that crawls and indexes web content.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Jobiglo-BotJobiglo-Bot is a web crawler operated by Jobiglo, used to collect job listings and employment-related information from websites for aggregation on their job search platfo…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
JooblebotJooblebot is a web crawler operated by Jooble, a job search engine platform. This bot discovers and indexes job postings to help job seekers find employment opportunities…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Kagibotkagibot is the web crawler for Kagi, a privacy-focused ad-free search engine. It crawls the web to build Kagi's independent search index to provide high-quality, personal…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Level9SearchBotLevel9SearchBot is a web crawler operated by Level 9, a company that provides website design and development services for credit unions and community banks. The bot crawl…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
LinespiderLinespider is a web crawler operated by LINE Corporation that provides search results for LINE services while complying with the Robots Exclusion Protocol.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
LohiSoftBotLohiSoftBot builds LohiSoft's search index by recording page URLs, titles, meta descriptions, headings, and link text.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →