Agents & Bots

AIWebIndex

AIWebIndex is a web crawler operated by Lyrenth that builds an AI-readable index of web content for AI systems.

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

ApifyBot

ApifyBot is a web scraping and data extraction crawler by Apify that collects website content for use in AI, LLMs, RAG, and automation workflows.

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

ApifyWebsiteContentCrawler

ApifyWebsiteContentCrawler is a web crawler by Apify that extracts and downloads full website content for use in AI, data analysis, and automation workflows.

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

Bravebot

Bravebot is a web crawler by Brave that indexes pages for Brave Search, providing search data and AI-optimized context to power chatbots, agents, and RAG pipelines.

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

Brightbot

Brightbot is a web data collection crawler by Bright Data that extracts and structures public website content at scale, providing AI-ready data for model training, RAG pi…

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

CragCrawler

CragCrawler is a web scraping bot operated by CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services. The bot is used…

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

Crawlspace

Crawlspace is a web crawler platform that fetches and extracts website content for AI agents, RAG applications, and structured data workflows.

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

Diffbot

Diffbot is a web crawler that extracts and structures website content using AI-powered visual understanding, providing knowledge graph data for applications like market i…

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

ExaBot

ExaBot is a web crawler that indexes web content to power Exa's AI search engine and semantic search APIs for AI applications.

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

FirecrawlAgent

FirecrawlAgent is a web crawler operated by Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications.

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

HenkBot

Henkbot crawls the web on behalf of Valyu, an AI search infrastructure provider that indexes content for use in AI-powered retrieval pipelines.

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

Mozilla-Tabstack

Mozilla-Tabstack is an AI agent operated by Mozilla that performs programmatic, AI-driven interactions with web content through Tabstack.

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

Querit-SearchBot

Querit-SearchBot is a web crawler operated by Querit that indexes web content for their search API service, which is designed to provide real-time search results for larg…

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

QueritBot

QueritBot is a web crawler operated by Querit, a company providing a search API for large language model integration. This bot indexes web content to power the real-time …

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

ShapBot

ShapBot is a web crawler by Parallel that collects and structures web content to power its search, extraction, and deep research APIs, providing AI agents with high-accur…

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

TavilyBot

TavilyBot is a web crawler by Tavily that indexes and extracts content from billions of pages, providing real-time search, extraction, and research data to ground AI agen…

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

Terra Cotta

Terra Cotta is Ceramic's web crawler that indexes public content to power their web-scale search API for AI and LLMs.

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

TerraCotta

TerraCotta is Ceramic's web crawler that indexes public content to power their web-scale search API for AI and LLMs.

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

YouBot

YouBot is a web crawler by You.com that indexes and extracts web content to power its real-time search, contents, and research APIs, delivering grounded web data to AI ag…

AI Data Provider

Crawls websites to supply structured content to AI systems as a third-party service

See More →

AI Data Providers