What is Google-Extended?
Google-Extended is a web crawler used by Google to download AI training content for its AI products like the Gemini assistant and its Vertex AI generative APIs.
Category
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
Do you operate this agent? Contact us to suggest an update.
| Operated By | |
| Source | Official Website |
| Used for AI Model Training | Yes |
| Affects AI Chat & Search Visibility | Not directly |
Google-Extended Should Not Be Visiting Websites
Google-Extended is a robots.txt control token that tells the operator how your data should be used. The actual crawling is done by other agents.
Should I Block Google-Extended?
Block Google-Extended if you want more control over whether your work is used for AI training. Allowing it may increase the chance that your brand appears in AI-generated answers, but either choice leaves traditional search rankings unchanged. For context, 20% of the top websites we track currently have robots.txt rules for Google-Extended.
How Do I Block Google-Extended Using Robots.txt?
Add this rule to your robots.txt file to block Google-Extended from accessing your website, or use Automatic Robots.txt to block all AI data scrapers at once. You can customize which pages are blocked by swapping out / for a different path.
User-agent: Google-Extended # https://knownagents.com/agents/google-extended
Disallow: /Other Google Agents
-
Google-AgentGoogle-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.AI AgentAI AgentUses an actual web browser to autonomously complete complex tasks on behalf of a human userSee More →
-
GoogleAgent-MarinerGoogleAgent-Mariner is an AI agent created by Google that can use a web browser. It can intelligently navigate and interact with websites to complete multi-step tasks on …AI AgentAI AgentUses an actual web browser to autonomously complete complex tasks on behalf of a human userSee More →
-
Gemini-Deep-ResearchGemini-Deep-Research is the agent responsible for collecting and scanning resources used in Google Gemini's Deep Research feature, which acts as a personal research assis…AI AssistantAI AssistantFetches website content in response to a user prompt, to include in an AI-generated answerSee More →
-
Google-NotebookLMGoogle-NotebookLM is an AI-powered research and note-taking assistant that helps users synthesize information from uploaded sources like documents, transcripts, or web co…AI AssistantAI AssistantFetches website content in response to a user prompt, to include in an AI-generated answerSee More →
-
GoogleAgent-URLContextGoogleAgent-URLContext is a web fetcher operated by Google that retrieves web content on behalf of Gemini API users. When a developer provides a URL as context in a Gemin…AI AssistantAI AssistantFetches website content in response to a user prompt, to include in an AI-generated answerSee More →
-
Google-Gemini-CLIGemini CLI is an AI coding agent by Google that can query and edit large codebases, generate apps from images or PDFs, and automate complex workflows directly from the te…AI Coding AgentAI Coding AgentFetches documentation and other resources to help build softwareSee More →
-
CloudVertexBotCloudVertexBot is a Google-operated crawler available to site owners to request targeted crawls of their own sites for AI training purposes on the Vertex AI platform.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
GoogleOtherGoogleOther is Google's generic crawler used by various product teams for fetching publicly accessible content, including one-off crawls for internal research and develop…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
Google-CloudVertexBotGoogle-CloudVertexBot is a web crawler operated by Google that collects content for Google Cloud's Vertex AI Search service. This crawler indexes web pages to power enter…AI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search resultsSee More →
-
Chrome-LighthouseChrome-Lighthouse is an automated, open-source tool that audits web pages for performance, accessibility, progressive web apps, SEO, and best practices. It runs a series …Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
Google Inspection ToolGoogle Inspection Tool is a developer helper operated by Google.Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
Google Schema Markup Testing ToolGoogle Schema Markup Testing Tool is a developer helper operated by Google.Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
Google Trust Services (DCV Check)Google Trust Services (DCV Check) is a developer helper operated by Google.Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
Google-Site-VerificationGoogle-Site-Verification is a verification service that proves website ownership for Google Search Console access by checking verification tokens like HTML tags or files …Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
Google-Structured-Data-Testing-ToolGoogle-Structured-Data-Testing-Tool is a developer helper bot operated by Google that validates structured data markup on web pages. It checks whether schema markup and r…Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
GoogleAssociationServiceGoogleAssociationService is a verification bot that checks Digital Asset Links statements on websites to validate relationships between web properties and mobile apps for…Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
ProjectShield Url CheckProjectShield Url Check is a developer helper operated by Google.Developer HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionalitySee More →
-
Google AppsViewerGoogle AppsViewer is a bot operated by Google that fetches and renders content from websites to provide previews or display embedded content within Google Drive and relat…FetcherFetcherRetrieves web page metadata to power app features like link previews or feedsSee More →
-
Google Feed FetcherGoogle Feed Fetcher is a fetcher operated by Google.FetcherFetcherRetrieves web page metadata to power app features like link previews or feedsSee More →
-
Google Image ProxyGoogle Image Proxy is a fetcher operated by Google.FetcherFetcherRetrieves web page metadata to power app features like link previews or feedsSee More →
-
Google Publisher CenterGoogle Publisher Center is a fetcher operated by Google.FetcherFetcherRetrieves web page metadata to power app features like link previews or feedsSee More →
-
Google Web PreviewGoogle Web Preview is a fetcher operated by Google.FetcherFetcherRetrieves web page metadata to power app features like link previews or feedsSee More →
-
Google-AdWords-ExpressGoogle-AdWords-Express is an automated crawler that analyzes advertiser websites to assist with Google Ads creation and campaign optimization, specifically designed for s…FetcherFetcherRetrieves web page metadata to power app features like link previews or feedsSee More →
-
Google-PageRendererGoogle-PageRenderer is a specialized tool used by Google services for rendering web pages and fetching content when triggered by end-user actions, supporting various Goog…FetcherFetcherRetrieves web page metadata to power app features like link previews or feedsSee More →
-
Google-Read-AloudGoogle-Read-Aloud fetches web content to provide text-to-speech functionality, enabling users to have web pages read aloud through Google's accessibility services.FetcherFetcherRetrieves web page metadata to power app features like link previews or feedsSee More →
-
GoogleDocsGoogleDocs is a fetcher bot operated by Google that visits websites when users embed or link to external content within Google Sheets and other Google Docs applications. …FetcherFetcherRetrieves web page metadata to power app features like link previews or feedsSee More →
-
GoogleImageProxyGoogleImageProxy fetches and caches images and link previews for Gmail email rendering, enabling safe display of external content within Google's email interface.FetcherFetcherRetrieves web page metadata to power app features like link previews or feedsSee More →
-
GoogleMessagesGoogleMessages is Google's fetcher that retrieves web page metadata to generate link previews for URLs shared in Google Messages.FetcherFetcherRetrieves web page metadata to power app features like link previews or feedsSee More →
-
GoogleProducerGoogleProducer is a specialized Google crawler that fetches and processes news feeds and content that publishers explicitly provide for display on Google News landing pag…FetcherFetcherRetrieves web page metadata to power app features like link previews or feedsSee More →
-
AdsBot-GoogleAdsBot-Google is a specialized Google crawler that monitors and analyzes websites where Google Ads are served to ensure quality and policy compliance.Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
AdsBot-Google-MobileAdsBot-Google-Mobile is a crawler operated by Google that specifically checks mobile web pages for ad quality purposes. The bot ensures ads display correctly on mobile de…Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
Google AdsBotGoogle AdsBot is an intelligence gatherer operated by Google.Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
Google AdSenseGoogle AdSense is an intelligence gatherer operated by Google.Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
Google-adstxtGoogle-adstxt is an intelligence gatherer operated by Google.Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
Google-Adwords-InstantGoogle-Adwords-Instant is the web crawler bot connected to the Google Ads platform that visits advertiser landing pages to perform quality checks, verification, and compl…Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
Google-Display-Ads-BotGoogle-Display-Ads-Bot is operated by Google and crawls websites to analyze their content for contextual ad targeting and placement in the Google Display Network. This bo…Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
Mediapartners-GoogleMediapartners-Google crawls website content to analyze page context and keywords for Google AdSense ad targeting, determining appropriate advertisements to display based …Intelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targetingSee More →
-
Feedfetcher-GoogleFeedfetcher-Google is a crawler operated by Google that fetches RSS and Atom feeds for various Google products and services. The bot retrieves syndication feeds to help i…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Google FaviconGoogle Favicon is a search engine crawler operated by Google.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Google ImagesGoogle Images is a search engine crawler operated by Google.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Google ScholarGoogle Scholar is a search engine crawler operated by Google.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Google VideosGoogle Videos is a search engine crawler operated by Google.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
GooglebotGooglebot is Google's web crawler that discovers and indexes web content for Google Search, including both mobile and desktop variants that crawl websites to understand t…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Googlebot-IAGooglebot-IA is Google Scholar's specialized crawler that indexes scholarly literature from academic publishers, repositories, and university websites to populate the aca…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Googlebot-ImageGooglebot-Image is Google's specialized search engine crawler that discovers and indexes images across the web for inclusion in Google Images search results, analyzing vi…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Googlebot-MobileGooglebot-Mobile is the mobile crawling component of Google's web crawler system used to index and analyze mobile versions of websites for inclusion in Google's mobile se…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Googlebot-NewsGooglebot-News is a search engine crawler operated by Google.Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Googlebot-VideoGooglebot-Video is Google's specialized crawler for video content discovery and indexing. It crawls websites to find and analyze video files, metadata, and thumbnails for…Search Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search resultsSee More →
-
Google-InspectionToolGoogle-InspectionTool powers Search Console's URL Inspection tool and Rich Results Test, crawling pages to validate indexability, structured data markup, and search featu…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
Google-SafetyGoogle-Safety is a specialized security crawler that performs abuse-specific scanning to detect malware and other security threats on publicly posted links across Google …Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
Google-Trust-ServicesGoogle-Trust-Services is a certificate authority bot that helps build a safer Internet by providing transparent, trusted, and reliable TLS certificates. It crawls website…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
ProjectShield-UrlCheckProjectShield-UrlCheck is a security scanner that monitors websites protected by Google's Project Shield service, checking for malicious traffic and DDoS attacks to ensur…Security ScannerSecurity ScannerScans websites for security vulnerabilities, threats, and configuration weaknessesSee More →
-
Google Digital Asset LinksGoogle Digital Asset Links is an SEO crawler operated by Google.SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
Google StoreBotGoogle StoreBot is an SEO crawler operated by Google.SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
Google-Structured-Data-TestingGoogle-Structured-Data-Testing bot validates structured data markup on web pages, helping developers ensure proper schema implementation for rich search results and SEO o…SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
Storebot-GoogleStorebot-Google is Google's web crawler that analyzes e-commerce websites, product pages, and checkout processes using machine learning to gather information on pricing, …SEO CrawlerSEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunitiesSee More →
-
AI2BotAI2Bot is operated by Ai2, a non-profit AI research institute. It's used to download data to train open source AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
Ai2Bot-DolmaAi2Bot-Dolma is operated by Ai2, a non-profit AI research institute. It's used to download data to train open source AI models.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
AmazonbotAmazonbot is a web crawler operated by Amazon that builds an index of web content to improve Amazon products and services, including Alexa, Kindle, and Amazon Shopping. T…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
Applebot-ExtendedApplebot-Extended is an Apple robots.txt control token that lets publishers opt out of using Applebot-crawled content to train Apple foundation models. It does not crawl …AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
BeansLLM-CorpusBotBeansLLM-CorpusBot collects public web content to build corpora for language-model training.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
bedrockbotbedrockbot is a web crawler operated by Amazon that crawls selected URLs for Amazon Bedrock knowledge bases and custom AI applications.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
BytespiderBytespider is a web crawler operated by ByteDance, the Chinese owner of TikTok. It's allegedly used to download training data for its LLMs (Large Language Model) includin…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
CCBotCCBot is Common Crawl's web crawler that creates an open repository of web data, making crawled content universally accessible for research, analysis, and AI training pur…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ChatGLM-SpiderChatGLM-Spider is a web crawler operated by Zhipu AI, the Chinese company behind ChatGLM. It is used for collecting data to train and evaluate the company's large languag…AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →
-
ClaudeBotClaudeBot is a web crawler operated by Anthropic to download training data for its LLMs (Large Language Models) that power AI products like Claude.AI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMsSee More →