Agent Directory
The definitive guide to every known AI agent, crawler, scraper, and other bot on the internet, updated daily.
Security Scanners
-
MerchantSecurityScannerMerchantSecurityScanner scans Stripe merchant websites for common security vulnerabilities, including exposed credentials and unprotected administrative pages. Stripe use…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
Netcraft SSL Server SurveyNetcraft SSL Server Survey is a security scanner operated by Netcraft, a cybersecurity and digital risk protection company. This bot scans websites to collect SSL/TLS cer…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
Nmap Scripting EngineNmap Scripting Engine is a component of Nmap, a free and open source network discovery and security auditing tool. It scans networks and hosts to determine available serv…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
ObserverObserver is a web security crawler that scans websites to identify third-party risks, security vulnerabilities, and performance issues. It detects compromised external re…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
OdinOdin is a web crawler operated by ODIN, an internet scanning search engine platform. The bot scans websites as part of indexing operations that power searches across vari…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
OneTrustOneTrust is a privacy compliance crawler that scans customer websites to identify and categorize cookies, tracking technologies, forms, tags, and storage mechanisms for p…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
OpenBash-SurfaceOpenBash-Surface performs automated external security checks for OpenBash, identifying exposed services, insecure configurations, and website vulnerabilities visible from…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
OsanoBotOsanoBot visits websites to evaluate privacy and compliance signals for Osano's automated compliance checks.Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
ProjectShield-UrlCheckProjectShield-UrlCheck is a security scanner that monitors websites protected by Google's Project Shield service, checking for malicious traffic and DDoS attacks to ensur…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
RedactusWebClassifierRedactusWebClassifier fetches website homepages for Redactus to classify sites and periodically check whether their content or category has changed.Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
Sansec Security MonitorSansec Security Monitor is a specialized e-commerce security scanner that monitors online stores for malware, vulnerabilities, and digital skimming attacks, providing rea…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
SecurityHeadersSecurityHeaders is a free security scanning service that analyzes HTTP security headers on websites, providing grades and recommendations to improve web security posture …Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
SiteLockSpiderSiteLockSpider is SiteLock's web scanning service that crawls websites to detect malware, malicious code, and security vulnerabilities for website protection.Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
Software Security ResearchSoftware Security Research is an academic research crawler operated by the Chair for Software Security at Ruhr University Bochum in Germany. This bot performs HTTP and HT…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
SSL LabsSSL Labs is Qualys' free online SSL/TLS security testing service that performs deep analysis of SSL web server configurations on the public Internet to identify security …Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
SucuriSucuri crawls websites to detect malware, security vulnerabilities, and blacklist status as part of its cloud-based website security platform and malware removal services…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
TermlyBotTermlyBot is Termly's web scanner that analyzes websites for legal compliance, privacy policy requirements, and cookie usage to ensure regulatory adherence.Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
TwinWaveScannerTwinWaveScanner (Splunk Attack Analyzer) uses headless Chrome browsers to analyze URLs for malicious content, performing automated threat analysis with comprehensive DOM,…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
UsercentricsbotUsercentricsbot is a security scanner operated by Usercentrics that analyzes websites for data protection and privacy compliance issues. The bot is part of their DPS (Dat…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
VerispiderVerispider is a security scanning bot that analyzes websites for spam and security threats, utilizing Project Honeypot IP intelligence for threat assessment.Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
visionheight.com/scanvisionheight.com/scan is a security scanner.Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
WTotemWTotem is a security monitoring bot that scans websites and web applications to detect malware, vulnerabilities, and other security threats.Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
WTotemBotWTotemBot is a security scanner operated by WebTotem, a website security monitoring service that provides firewall protection, antivirus scanning, and vulnerability manag…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
SEO Crawlers
-
aaaa is a web crawler used by Agency Analytics for collecting SEO data and analyzing website performance to help marketing agencies generate client reports.SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
Ahrefs Site AuditAhrefs Site Audit is an SEO crawler operated by Ahrefs.SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
AhrefsBotAhrefsBot is Ahrefs' web crawler that powers both the Ahrefs marketing intelligence platform and Yep search engine, constantly crawling the web to build a comprehensive l…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
AhrefsSiteAuditAhrefsSiteAudit is a technical SEO crawler that analyzes websites to identify and flag over 170 technical and on-page SEO issues, providing detailed recommendations and p…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
AttractaAttracta is a web crawler that analyzes website content and structure to provide SEO optimization services, including link building strategies and search engine ranking i…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
Audisto CrawlerAudisto Crawler systematically fetches all accessible URLs of websites to provide comprehensive site auditing and monitoring services for SEO and technical analysis.SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
BarkrowlerBarkrowler is an SEO web crawler operated by Babbar that analyzes websites to identify page popularity, topical relevance, and authority metrics for search engine optimiz…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
BLEXBotBLEXBot is WebMeUp's web crawler that builds the world's freshest backlink index for SEO PowerSuite Link Explorer, providing comprehensive backlink data for SEO analysis …SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
botifybotify's SiteCrawler analyzes website structure and content like a search engine, giving enterprise SEO teams detailed insights into how their sites are crawled and index…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
BrightEdge CrawlerBrightEdge Crawler powers the Autopilot SEO automation tool, crawling customer websites to analyze internal linking opportunities and identify images needing optimization…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
CaliberBotCaliberBot is an SEO crawler operated by Conductor.SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
CaliperbotCaliperbot is a web crawler that analyzes website content and technical features for Conductor's SEO intelligence platform, extracting HTML elements and data to power con…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
ClarityBotClarityBot is an SEO crawler operated by seoClarity, a search engine optimization platform. This bot crawls websites to collect data for SEO analysis and insights, helpin…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
ClearscopebotClearscopebot crawls content for Clearscope, an AI-powered SEO content optimization platform by Mushi Labs. Analyzes top-performing content to provide keyword recommendat…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
Cloudflare Custom Hostname VerificationCloudflare Custom Hostname Verification is an SEO crawler.SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
Cloudflare Stream WebhookCloudflare Stream Webhook is an SEO crawler.SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
Cloudflare-HealthchecksCloudflare-Healthchecks is an SEO crawler.SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
CloudFlare-PrefetchCloudFlare-Prefetch is a bot operated by Cloudflare that prefetches web content to improve page load times and performance for websites using Cloudflare's content deliver…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
Cloudflare-SSLDetectorCloudflare-SSLDetector is an SEO crawler.SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
Cloudflare-Traffic-ManagerCloudflare-Traffic-Manager is an SEO crawler.SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
CloudtrellisCloudtrellis automatically scans entire websites for broken links, accessibility issues, SEO problems, and security risks to ensure optimal user experience and search eng…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
cludo.comcludo.com is the web crawler bot that indexes and analyzes website content to provide AI-driven site search solutions, search analytics, and content optimization for over…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
CocolyzebotCocolyzebot is an SEO crawler operated by Cocolyze, an SEO tool software company. The bot crawls websites to identify technical errors, analyze content quality and loadin…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
cognitiveSEO CrawlercognitiveSEO Crawler is an SEO crawler operated by cognitive SEO Internet Marketing Tools.SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
contentkingcontentking is a real-time SEO monitoring crawler that continuously scans websites to detect technical SEO issues, content changes, and performance problems for digital m…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
ConvermaxConvermax Site Search Indexer crawls auto parts e-commerce websites to index product data and fitment information for specialized automotive search functionality and Year…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
CxenseCxense is an SEO crawler operated by Cxense.SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →