How to tell if ChatGPT reads your page: server log signals explained (2026)
AI assistants leave traces in your server logs, but not every trace means the same thing. Some bots collect training data, some build a search index, and some fetch a page because a person asked a question at that moment. That last group is the closest you get to evidence that an assistant used your page in a live answer.
This guide explains how to tell these visits apart, how to verify them, and which mistakes we made in our own logs.
The short answer
- Look for user-triggered fetches:
ChatGPT-User,Claude-UserandPerplexity-User. These agents visit a page when a question in the assistant needs it. - Search index crawlers (
OAI-SearchBot,Claude-SearchBot,PerplexityBot) decide whether you can be found at all. Training crawlers (GPTBot,ClaudeBot) tell you nothing about live answers. - Never trust the user agent alone. Filter on pages that exist and on successful status codes, and check the IP address against the ranges the provider publishes.
Three kinds of AI bot visits
| Provider | Training | Search index | User-triggered fetch |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | not listed | PerplexityBot | Perplexity-User |
| Google-Extended (a robots.txt token, not a separate visitor) | Googlebot | not listed |
What the providers document, in short:
- OpenAI describes ChatGPT-User as an agent for user actions that does not crawl automatically and does not decide whether content can appear in ChatGPT search. Because these visits start with a user, robots.txt rules may not apply. Search visibility is controlled through OAI-SearchBot. OpenAI notes that a robots.txt change can take about 24 hours to affect search.
- Anthropic describes Claude-User as the agent that retrieves pages when someone asks Claude a question, and Claude-SearchBot as the agent that indexes content for search. Blocking either one reduces visibility in user search. Anthropic says its bots honor robots.txt and the Crawl-delay extension.
- Perplexity describes PerplexityBot as the crawler that surfaces and links websites in its search results, and Perplexity-User as the agent for user actions.
The signal that matters most: user-triggered fetches
A ChatGPT-User request means a ChatGPT session needed your page at that moment. That is a stronger signal than an index crawl. It is not proof of a citation: an assistant can fetch a page and then not use it.
What we saw on our own site: our English comparison of AI visibility tools received 26 ChatGPT-User fetches on 14 different days between 14 September and 28 October 2026. The Dutch version of the same page, with the same structure, received none in that period.
During the same weeks, our measurements through the OpenAI API with an older model never cited that page, because the model rarely searched the web. The logs showed real use that an API measurement missed.
How to check your own logs
The examples below assume the common nginx or Apache "combined" log format. Check the time zone offset in your log lines (for example +0000) before you compare times with anything else.
1. Count visits per agent
grep -Eo "ChatGPT-User|OAI-SearchBot|GPTBot|Claude-User|Claude-SearchBot|ClaudeBot|Perplexity-User|PerplexityBot" access.log | sort | uniq -c | sort -rn
For rotated, compressed logs, use zgrep on the .gz files.
2. See which pages a user-triggered agent fetched successfully
grep "ChatGPT-User" access.log | awk '$9 == 200 {print $7}' | sort | uniq -c | sort -rn | head -20
In the combined format, field 9 is the status code and field 7 is the requested path.
3. Follow one page per day
grep "ChatGPT-User" access.log | grep " /en/your-page " | awk '{print substr($4, 2, 11)}' | sort | uniq -c
4. Verify the IP addresses
OpenAI and Perplexity publish IP ranges per agent as JSON files (see the sources below). Anthropic refers to an IP list on its crawler page. This short Python script checks whether an IP address falls inside a published list, regardless of how the JSON is structured:
import ipaddress
import json
import sys
import urllib.request
def published_networks(url):
request = urllib.request.Request(url, headers={"User-Agent": "verify-ip/1.0"})
data = json.load(urllib.request.urlopen(request))
networks = []
def walk(value):
if isinstance(value, dict):
for item in value.values():
walk(item)
elif isinstance(value, list):
for item in value:
walk(item)
elif isinstance(value, str):
try:
networks.append(ipaddress.ip_network(value, strict=False))
except ValueError:
pass
walk(data)
return networks
url, ip = sys.argv[1], ipaddress.ip_address(sys.argv[2])
print(any(ip in network for network in published_networks(url)))
Run it as python3 verify_ip.py https://openai.com/chatgpt-user.json 203.0.113.7.
Five pitfalls we ran into
1. Scanners wearing an AI user agent. Between 12 May and 13 August 2026, 98.9% of requests with a Perplexity-User user agent on our site went to paths that do not exist, such as .env and .git/config. That is vulnerability scanning, not an assistant. Filter on existing paths and successful status codes, and verify the IP.
2. Your own tools. Monitoring tools and access checks use the same user agents. On our site, every successfully served Perplexity-User request (existing pages, status 200, 206 or 304) between 1 August and 25 October 2026 came from our own access check. Exclude your own traffic by IP address and by time window.
3. Developer tools. Coding assistants can fetch pages with a Claude-User user agent plus an extra marker such as claude-code. That is someone using a developer tool, not an answer in Claude. Count these requests separately.
4. Loose pattern matching. Match the full product token, for example Claude-SearchBot/1.0, not a fragment elsewhere in the string. The Anthropic user agents in our logs contain the contact address claudebot@anthropic.com. Between 6 May and 29 October 2026, a pattern in our own dashboard that matched on that word counted 1,027 requests with a Claude-SearchBot or Claude-User user agent as the training crawler ClaudeBot. Almost all of them asked for paths like .env and .git/config and received an error: scanners using an Anthropic user agent. After filtering on existing pages and successful status codes, 5 remained.
5. Order is not cause. On our comparison page, the first OAI-SearchBot request and the first ChatGPT-User request arrived in the same second. Logs show that both happened, not which one caused the other.
What logs cannot tell you
- Whether the assistant cited or mentioned you in its answer.
- Which question the user asked.
- Visits from assistants that do not identify themselves.
Logs show whether assistants reach your pages. Whether they use them in answers requires measuring the answers themselves. That is what Lens does, including an access check per bot category for your site.
How we measured
- Source: nginx access logs of hiveminds.nl, times in UTC.
- Filters: existing paths only, status 200, 206 or 304, our own monitoring and test traffic excluded, developer tools counted separately.
- Periods: stated with each figure.
- Provider documentation: checked on 29 October 2026.