How AI search finds your site: live

Live bot-traffic data from two sites in our portfolio, updated every minute. No mockup, no marketing, just the logs.

Reach

AI bots index your site. Sitemap, robots.txt, content. Search engines, but with different intent.

Read

What they fetch, how often, which pages. This data is normally hidden in nginx logs.

Cite

Which content comes back in answers? That's what Lens is about.

→ /lens

Handler.gg: what AI search does at scale

A padel-aggregator with national coverage. Here you see how AI bots treat an optimised site.

Bot visits (24h)6289
AI bot visits (24h)4556
AI vs search ratioAI = 3.8× search
Live feed
0s ago updated
27s agoBingbot/nl/padelbanen/velp200
1m agofacebookexternalhit/301
2m agoBaiduspider/assets/city-en-BiBna-19.js200
2m agoBaiduspider/en/padel-courts/brakel200
3m agofacebookexternalhit/301
5m agoGooglebot/nl/padelbanen/laren/plaza-padel-laren200
5m agofacebookexternalhit/301
7m agoGooglebot/nl/padelbanen/provincie/noord-brabant200
7m agoGooglebot/fr/terrains-de-padel/sint-gillis-waas200
7m agoGooglebot/nl/padelbanen/utrecht200
7m agoGooglebot/en/padel-courts/province/drenthe200
7m agoGooglebot/nl/padelbanen/provincie/overijssel200
7m agoGooglebot/en/padel-courts/groningen200
7m agoGooglebot/nl/padelbanen/provincie/gelderland200
7m agoGooglebot/nl/hoe-het-werkt200
7m agoGooglebot/nl/padelbanen/provincie/drenthe200
7m agoGooglebot/fr/terrains-de-padel/sint-niklaas/the-padel-point200
7m agoGooglebot/fr/terrains-de-padel/stekene200
7m agoGooglebot/fr/terrains-de-padel/beveren200
7m agoGooglebot/nl/padelbanen/arnhem200
Bot traffic by category
Top 10 bots (24h)
BotCategoryVisits ↓% totalVerifiedLast seen
ClaudeBotAI4.19466.7%100%1h ago
BaiduspiderSearch6049.6%n/a2m ago
ApplebotSearch3866.1%n/a2h ago
facebookexternalhitSocial3135.0%n/a1m ago
SemrushBotSEO2093.3%n/a18m ago
AmazonbotAI1682.7%n/a9m ago
GooglebotSearch1121.8%n/a5m ago
OAI-SearchBotAI1081.7%100%2h ago
BingbotSearch841.3%n/a27s ago
GPTBotAI410.7%100%2h ago

Hiveminds.nl: active AI crawlers and agents

Which bots read this site? We see active crawlers and agents from major AI and search platforms: OpenAI, Anthropic, Perplexity, Google, Meta and ByteDance. No coincidence.

ClaudeBot
52 visits (24h)of which 52 verified14m ago
Meta-ExternalAgent
34 visits (24h)1m ago
Bytespider
30 visits (24h)2h ago
Amazonbot
25 visits (24h)2h ago
ChatGPT-User
8 visits (24h)of which 8 verified9h ago
OAI-SearchBot
5 visits (24h)of which 5 verified2h ago
GPTBot
1 visits (24h)of which 1 verified10h ago
Monitored AI crawlersGPTBot (OpenAI) · ClaudeBot / anthropic-ai (Anthropic) · PerplexityBot (Perplexity)

Crawl history

Bot crawls per day on hiveminds.nl. Vertical line marks the switch to server-side rendering.

Crawls
GPTBot
ClaudeBot
PerplexityBot
Googlebot
Bingbot
Other AI
Other

Counted by user agent until 31 October 2026, with an IP check per request from 31 October.

What content draws AI bots?

The most requested URLs on this site over the past 30 days. The pattern is predictable, and that is exactly the point.

/robots.txt505 hits317 verified6 bots
/sitemap.xml263 hits170 verified4 bots
/en/lens/ai-visibility-tools-comparison-202690 hits46 verified6 bots
/84 hits32 verified9 bots
/en31 hits11 verified7 bots
/assets/static/plus-jakarta-sans-latin-700-normal.CfpNZvy6.woff227 hits2 verified3 bots
/assets/static/jetbrains-mono-latin-ext-300-normal.tseANUoC.woff225 hits2 verified3 bots
/assets/static/plus-jakarta-sans-latin-ext-600-normal.DQgPU_Cg.woff224 hits2 verified3 bots
/assets/static/plus-jakarta-sans-vietnamese-400-normal.Cg2kxhWu.woff223 hits2 verified3 bots
/assets/static/plus-jakarta-sans-latin-ext-300-normal.PkbsTXxT.woff223 hits2 verified3 bots
/assets/static/plus-jakarta-sans-vietnamese-500-normal.DiU8zqi-.woff223 hits2 verified3 bots
/assets/static/plus-jakarta-sans-vietnamese-800-normal.CxbopLd3.woff223 hits1 verified2 bots
/assets/static/plus-jakarta-sans-vietnamese-600-normal.Y4VTTRme.woff222 hits2 verified3 bots
/assets/static/plus-jakarta-sans-latin-400-normal.Dhut76fR.woff222 hits1 verified2 bots
/assets/static/jetbrains-mono-latin-400-normal.V6pRDFza.woff222 hits1 verified2 bots

This list only counts requests the site actually answered. Traffic rejected at the server door, such as scanners probing for configuration files, does not count as a crawl. What you see here is requested content, not noise.

Methodology

Data comes directly from nginx access logs. We identify bots by their user agent. Since 31 October 2026, we also check requests from OpenAI, Anthropic and Perplexity against the IP ranges these companies publish themselves. A request is verified if its IP address falls within the list for that exact bot. Anthropic publishes one list for all its bots: we check ClaudeBot against it, while Claude-User and Claude-SearchBot do not fall within it in our logs and are therefore marked as cannot be checked. Other bots, such as Googlebot, are not checked yet. We do not store IP addresses, only the outcome of the check. Updated every minute. Only requests the site answered on a path that exists here, or once did, are counted; scanner traffic to invented URLs is not. How strictly you draw that line shifts the comparison by a few percentage points; the exact variants are in the measurement report.

Read more