Live bot-traffic data from two sites in our portfolio, updated every minute. No mockup, no marketing, just the logs.
AI bots index your site. Sitemap, robots.txt, content. Search engines, but with different intent.
What they fetch, how often, which pages. This data is normally hidden in nginx logs.
A padel-aggregator with national coverage. Here you see how AI bots treat an optimised site.
Which bots read this site? We see active crawlers and agents from major AI and search platforms: OpenAI, Anthropic, Perplexity, Google, Meta and ByteDance. No coincidence.
Bot crawls per day on hiveminds.nl. Vertical line marks the switch to server-side rendering.
Counted by user agent until 31 October 2026, with an IP check per request from 31 October.
The most requested URLs on this site over the past 30 days. The pattern is predictable, and that is exactly the point.
This list only counts requests the site actually answered. Traffic rejected at the server door, such as scanners probing for configuration files, does not count as a crawl. What you see here is requested content, not noise.
Data comes directly from nginx access logs. We identify bots by their user agent. Since 31 October 2026, we also check requests from OpenAI, Anthropic and Perplexity against the IP ranges these companies publish themselves. A request is verified if its IP address falls within the list for that exact bot. Anthropic publishes one list for all its bots: we check ClaudeBot against it, while Claude-User and Claude-SearchBot do not fall within it in our logs and are therefore marked as cannot be checked. Other bots, such as Googlebot, are not checked yet. We do not store IP addresses, only the outcome of the check. Updated every minute. Only requests the site answered on a path that exists here, or once did, are counted; scanner traffic to invented URLs is not. How strictly you draw that line shifts the comparison by a few percentage points; the exact variants are in the measurement report.