Live bot-traffic data from two sites in our portfolio, updated every minute. No mockup, no marketing, just the logs.
AI bots index your site. Sitemap, robots.txt, content. Search engines, but with different intent.
What they fetch, how often, which pages. This data is normally hidden in nginx logs.
A padel-aggregator with national coverage. Here you see how AI bots treat an optimised site.
Which bots read this site? We see active crawlers and agents from major AI and search platforms: OpenAI, Anthropic, Perplexity, Google, Meta and ByteDance. No coincidence.
Bot crawls per day on hiveminds.nl. Vertical line marks the switch to server-side rendering.
The most requested URLs on this site over the past 30 days. The pattern is predictable, and that is exactly the point.
This list only counts requests the site actually answered. Traffic rejected at the server door, such as scanners probing for configuration files, does not count as a crawl. What you see here is requested content, not noise.
Data comes directly from nginx access logs. Bots are identified by user-agent. Updated every minute. Only requests the site answered on a path that exists here, or once did, are counted; scanner traffic to invented URLs is not. How strictly you draw that line shifts the comparison by a few percentage points; the exact variants are in the measurement report.