Agent Directory
The definitive guide to every known AI agent, crawler, scraper, and other bot on the internet, updated daily.
Search Engine Crawlers
-
LuxxlebotLuxxlebot is the primary web crawler for Luxxle, a search engine that focuses on providing search results with minimal ads and enhanced privacy. This bot scans websites b…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
lyonllyonl is a search engine crawler operated by Lyonl.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
lyonl-crawlerlyonl-crawler is the web crawler for Lyonl Search, a search engine that indexes web content to provide search results. This bot crawls websites to build and maintain thei…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
MagiBotMagiBot is a web crawler operated by Peak Labs that focuses on information extraction and retrieval technology, collecting web content to transform natural language knowl…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Marginalia SearchMarginalia Search is a search engine crawler operated by Marginalia.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Mars FinderMars Finder is a search engine crawler operated by Mars Flag.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
MetaCrawlerMetaCrawler fetches public web pages for MetaCrawler's search service.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
MojeekBotMojeekBot is Mojeek's web crawler that indexes websites for their independent search engine, providing privacy-focused search results without tracking users.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
MotoMinerBotMotoMinerBot crawls and indexes vehicle detail pages from automotive websites to build MotoMiner's searchable vehicle database and search engine platform.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
MRGbotMRGbot is a search engine crawler operated by MRG Web Services srl, designed to generate a comprehensive corpus of data for advanced aggregation and analysis across multi…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
MSNMSN is a search engine crawler operated by Microsoft.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
msnbotmsnbot is Microsoft's legacy web crawler that indexed web pages for the MSN Search engine, which has since been replaced by Bing's crawling infrastructure, though some re…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
msnbot-mediamsnbot-media is a web crawler operated by Microsoft that specifically focuses on indexing media content for MSN Search. This bot discovers and collects images, videos, an…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
mwmblmwmbl is the web crawler for Mwmbl, an open source, non-profit search engine. It crawls websites to build and maintain the search index for the Mwmbl search engine, which…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
NagetBotNagetBot is a search engine crawler operated by Naget that crawls public pages for the Naget search index.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
NeevabotNeevabot is Neeva's web crawler that indexes websites for their ad-free search engine, which was designed to provide personalized search results without tracking users.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
NestDaddyBotNestDaddyBot is a search engine crawler operated by NestDaddy that indexes web content for their search engine.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
NewsNowNewsNow is a web crawler that aggregates news headlines from thousands of sources across the web to provide comprehensive, real-time news coverage through the NewsNow new…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
NotabotNotabot is a web crawler that integrates with Helpfeel's AI-powered FAQ search system, indexing website content to provide intelligent search and auto-complete capabiliti…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
PageSourceSearchBotPageSourceSearchBot crawls public pages and first-party JavaScript to build PageSourceSearch's searchable index of website source code.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
PublicWWWBotPublicWWWBot crawls website source code for PublicWWW. Its index lets users search HTML, JavaScript, and CSS for code snippets, technologies, analytics identifiers, and o…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
PurityBotPurityBot crawls public pages to build the independent index for Purili's privacy-focused search engine.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
QwantbotQwantbot is the web crawler for Qwant, the French privacy-focused search engine. Based in Europe, it indexes websites while prioritizing user privacy and data protection,…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Qwantbot-officialQwantbot-official is the web crawler for Qwant, a European search engine that indexes web pages to include them in search results. This bot visits websites to discover an…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
QwantifyQwantify is a search engine crawler operated by Qwant.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
ScrubbyScrubby is the web crawler for Scrub the Web, an SEO-friendly business directory that has been operating since 1996. The bot crawls submitted websites to index content an…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
search.marginalia.nusearch.marginalia.nu is a web crawler for Marginalia Search, a noncommercial search engine that focuses on indexing old websites, personal websites, and blogs. The search…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
SeekportSeekport is a search engine crawler operated by SISTRIX.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
SeekportBotSeekportBot is the web crawler for Seekport, an independent search engine operated by SISTRIX since 2014 that indexes web content to provide privacy-focused search result…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
SemanticScholarBotSemanticScholarBot is a search engine crawler operated by Ai2.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
SeznamSeznam is a search engine crawler operated by Seznam.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
SeznamBotSeznamBot is Seznam's web crawler that indexes websites for their Czech search engine, providing localized search results primarily for Czech and Slovak users.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
SiteSearch360SiteSearch360 crawls and indexes website content to provide custom site search functionality, offering semantic search capabilities as a Google Site Search replacement.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
slopsearch-macos-crawlerslopsearch-macos-crawler crawls and indexes public documents for a private search-engine research project.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
SMTnetPMBotSMTnetPMBot crawls partner company websites to index and include their content in SMTnet's on-site search engine for electronics manufacturing industry resources.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Sogou Push SpiderSogou Push Spider is a web crawler operated by Sogou, a Chinese search engine company. This bot crawls and indexes web pages for inclusion in Sogou's search results, serv…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Sogou web spiderSogou web spider is the web crawler operated by Sogou, a Chinese search engine company. This bot discovers and indexes content for Sogou's search results to build and mai…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
SokoCrawlerSokoCrawler is a web crawler operated by eXperience Tech for the Istrazi.rs search engine. It indexes web content to populate search results for users of the Istrazi.rs p…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
stepstoneCrawlBotstepstoneCrawlBot is a web crawler operated by The Stepstone Group, a digital recruitment platform. This bot crawls websites to discover and index job postings and career…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
TeomaTeoma is a search engine crawler operated by Ask.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
TinEyeTinEye is a search engine crawler operated by TinEye.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
TinEye-botTinEye-bot is a search engine crawler operated by TinEye.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
TinEye-bot-liveTinEye-bot-live is the web crawler operated by TinEye, a reverse image search engine. This bot indexes images from websites to help users find where images appear online,…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
TinEye-WebTinEye-Web is a web crawler operated by TinEye, a reverse image search engine. This bot indexes images and their locations across the internet to power TinEye's reverse i…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
VivianCrawlerVivianCrawler is a web crawler operated by Evil Source Platform that indexes public web content for search.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
WepchSearchEngineWepchSearchEngine is a web crawler associated with Wepch, a search engine project that is currently in development but not yet launched. The crawler is used to index and …Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
What10BotWhat10Bot crawls public web pages to build the index for What10's independent search engine.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Y!J-BRZ/YATSHA crawlerY!J-BRZ/YATSHA crawler is a search engine crawler operated by Yahoo Japan that indexes web pages for inclusion in Yahoo Japan's search results. This bot visits websites t…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
yacybotyacybot is the web crawler for YaCy, a decentralized peer-to-peer search engine that operates without central servers. The bot indexes content for the YaCy network, enabl…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Yahoo! SlurpYahoo! Slurp crawls and indexes web content for Yahoo's search engine, collecting data to power search results and web discovery features.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →