Real-time bot-traffic data from two sites in our portfolio. No mockup, no marketing — just the logs.
AI bots index your site. Sitemap, robots.txt, content. Search engines, but with different intent.
What they fetch, how often, which pages. This data is normally hidden in nginx logs.
A padel-aggregator with national coverage. Here you see how AI bots treat an optimised site.
Which AI search engines crawl this site? All the big ones: ChatGPT, Claude, Perplexity, OpenAI Search, Google's AI tier, Meta, ByteDance. No coincidence.
Daily bot crawls on hiveminds.nl. Vertical line marks the switch to server-side rendering.
The top crawled URLs on this site. The pattern is predictable — and that is exactly the point.
See what comes up here? Sitemap files, content about AI search itself, schema markup explainers. Bots seek what they understand — structured, explicit content. That's not coincidence, that's design.
Data comes directly from nginx access logs. Bots are identified by user-agent. Updated every minute. Our own IP is excluded from the count.