AI crawlers vs. human visitors vs. search bots: how to tell the traffic apart
Not every automated request is the same kind of visitor, and not every bot can be identified from a log line.
Open a raw server log and you will see lines from browsers, from search engines, from monitoring tools, from scrapers, and now from the crawlers that belong to AI products. They all look like HTTP requests. They are not the same kind of visitor, and mixing them up leads to bad conclusions, such as "a bot hit our pricing page, so buyers are asking about us."
This article explains the main categories of visitor, what each one is for, and the limits of what a log can tell you.
Human visitors
A person using a browser loads a page, then the page's images, scripts, and styles. A typical human session shows a sequence of related requests: the HTML, then assets, then perhaps a click to another page. Browsers send a user-agent string that names the browser, and they usually execute JavaScript, send cookies, and carry a referrer.
Analytics tools that rely on JavaScript see these visits. They usually do not see simple crawlers that fetch only HTML.
Search and answer crawlers
Operators of search products run crawlers that fetch pages to build an index. Several AI products now run their own, and their operators publish documentation. For example, OpenAI documents OAI-SearchBot as the crawler used to surface sites in ChatGPT search features, and Anthropic documents Claude-SearchBot as the crawler that improves the quality of Claude's search results. OpenAI describes its crawler settings as independent of one another, and Anthropic lists its crawlers separately with a different purpose for each.
These crawlers fetch on their own schedule, fetch many pages, and, according to their operators, follow robots.txt.
Training crawlers
Training crawlers collect content that may be used to train models. OpenAI documents GPTBot for this purpose, and Anthropic documents ClaudeBot. Blocking a training crawler is a publisher choice. According to the operators' own descriptions, it is a separate decision from whether you appear in their search or answer features, so a robots.txt rule for one should not be read as a decision about the other.
User-triggered fetchers
Some requests happen because a person asked an assistant to open a page. OpenAI documents ChatGPT-User for this and notes that, because the request is initiated by a user, robots.txt rules may not apply. Anthropic documents Claude-User as accessing sites when a user's question requires it, and says its bots honor robots.txt.
The practical point: a user-triggered fetch is evidence that someone asked about something that led the assistant to your page. It is not evidence about what they wanted, who they are, or whether they became a customer.
Why the distinction matters
| If you see | A reasonable reading | An unreasonable reading |
|---|---|---|
| Many requests from a documented search crawler | The operator is indexing your pages | "We are being recommended more" |
| A single user-triggered fetch of one page | Someone's question reached that page | "A buyer is evaluating us" |
| A training crawler on your blog | Content is being collected for training | "We will be cited in answers" |
| An unknown agent with an AI-sounding name | Unverified | "That is OpenAI" |
What logs can and cannot prove
A user-agent string is a claim, not proof. Anyone can send a request with any user-agent. Some operators publish IP address lists you can compare against, and Anthropic says a crawler whose source IP is on its published list comes from Anthropic. If you cannot verify, label the traffic "claims to be X."
You cannot identify every AI agent. New agents appear, some do not announce themselves, and some browse like ordinary browsers. A log analysis will always leave an "unknown" bucket. Do not hide it.
Visits do not imply recommendations. A crawler fetching a page tells you the page was requested. It does not tell you the page will be used in an answer, cited, or ranked.
Absence is weak evidence. A crawler not appearing in a log may mean it did not visit, but it can also mean the log is incomplete (a CDN cache served the request, logs were sampled, or the window was short).
A worked example (illustrative)
A team exports 14 days of access logs for their site. They classify 40,000 requests:
| Class | Requests | Notes |
|---|---|---|
| Browser-like, with JavaScript assets requested | 28,000 | Likely human sessions |
| Documented search or answer crawlers | 3,000 | Matched by user-agent; IP check pending |
| Training crawlers | 1,200 | Matched by user-agent |
| Known search-engine crawlers | 4,800 | Matched and verified |
| Unknown or unclassified | 3,000 | Left unlabelled |
The honest summary is: "Of 40,000 requests, 7.5% claim to be an AI search or answer crawler. We have not verified source IPs. 7.5% could not be classified at all." They do not conclude anything about buyer intent.
A safe way to read logs
- Pick a window long enough to be meaningful, such as two to four weeks, and say which one you used.
- Group by claimed user-agent first, then check against operator-published IP lists where available.
- Keep an "unknown" class.
- Separate pages by purpose (docs, pricing, blog) before comparing counts.
- Compare like with like. A crawler that fetches robots.txt on every visit will skew raw counts.
- Never infer intent from a single request.
How Doclight helps
Doclight's website audit crawls your site and evaluates robots.txt for documented answer-surface crawler names, showing the exact rule lines and the pages affected. Doclight also has an events endpoint where your own pipeline can send bot-visit events with an API key, and its dashboard shows counts by the bot type you send. Doclight does not read your server logs itself, does not verify that a user agent is who it says it is, and does not infer intent from bot visits. Treat those counts as your own classification, only as good as the pipeline that produced it.
Sources and verification
Operator descriptions were checked on 2026-10-12: OpenAI's overview of its crawlers (read through a search summary, as the page was not directly reachable from the research environment) and Anthropic's crawler support article (read directly). Crawler names and behavior change; check the operators' pages before acting. The log table is illustrative.
Related: what robots.txt can tell you about AI crawler access.