The Agentic Web Index

Traffic by Agent Type

Hover over each agent type for more information about what they do

Spoofed Traffic by Agent Identity

The percentage of impersonated website traffic for each agent identity over time

Traffic by AI Chat Platform

Referral activity by AI chat platform over time

Data Scope

The Index updates daily using completed days of traffic, security, and referral data from more than 5,000 websites using Agent Analytics and AI Chat Referral Tracking. Percentage changes compare equal-length periods. Agent data comes from the Agent Directory, and website categories follow Google AdSense. Participating websites are not a random sample, and the qualifying set changes over time, so results show observed directional trends rather than a census of global web traffic.

Qualification & Aggregation

Only website-days meeting minimum activity and data-quality requirements are included; internal, test, incomplete, and anomalous data is excluded. Bot percentages use all visits, while AI chat referral percentages use estimated human visits after identified bots are removed. Rates are calculated per website-day and then averaged, giving each website equal weight regardless of traffic. Daily charts are not smoothed in order to preserve natural seasonality.

Measuring Robots.txt Effectiveness

Robots.txt Effectiveness estimates the traffic reduction associated with blocking an agent sitewide using Disallow: /. For each agent and day, the baseline allowed rate includes every qualifying website that allows it, even those with zero agent visits:

For each blocked website, expected visits are:

Given observed disallowed visits O, its score is:

A 0% score means visits met or exceeded expectations, 50% means half as many visits as expected, and 100% means none were observed.

Scores require meaningful activity across multiple allowed and blocked websites over at least 7 of the latest 14 completed days. Each qualifying website-day has equal weight, and the headline score gives each qualifying agent equal weight. Only verified visits count for agents supporting authentication. This observational score measures outcomes, not intent or causation.

Top Blocked Bots is calculated separately from robots.txt scans of Similarweb's top 1,000 websites.

Identifying Spoofed Bots

Spoofing statistics count visits that claim a known agent's identity but fail a supported authentication method, such as IP verification or HTTP message signatures (Web Bot Auth). Each agent's daily rate is calculated against total visits per website and then averaged across websites. A failure suggests impersonation but does not identify the actual software or operator. Agents without supported authentication are excluded.

AI Chat Citations & Referrals

AI chat referrals count observed human visits with a recognized AI platform in the referrer or campaign source; visits without that data cannot be attributed. Citation estimates use requests from agents that retrieve content for AI platforms. Those requests may inform a response but do not prove a user saw a citation, so citation results are directional rather than exact counts.