Finding AI Crawlers in Your Server Logs: Who Visits, How Often, and How to Verify Them

Industry measurements now agree on something that would have sounded absurd five years ago: automated traffic has passed human traffic on the web. We looked at the big picture of that shift in bot traffic now exceeds half the web. This piece is the practical sequel, because the interesting question for a site owner is not “is the web full of bots” but “which of them are on my site, and what are they doing there?” There is exactly one place where that question gets answered honestly: your server logs. Analytics will not show these visitors, because AI crawlers do not run JavaScript tags. The log line is often the only trace they leave.

So let us read the logs. This is a guide to finding AI crawlers in your access log, verifying that they are who they claim to be, and thinking clearly about what their presence means.

Who You Are Looking For

AI-related crawlers fall into three groups that behave very differently, and mixing them up leads to bad decisions.

The first group is training crawlers, which collect content to train future models: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl, whose corpus many labs train on), Bytespider (ByteDance), meta-externalagent (Meta), Amazonbot (Amazon). Blocking these affects whether your content ends up inside a model’s weights someday; it does not affect whether you appear in AI search answers today.

The second group is search-index crawlers for AI answer engines: OAI-SearchBot (OpenAI’s search index) and PerplexityBot (Perplexity’s index). These decide whether your pages can be found and cited by those products. Blocking them is much closer to blocking Googlebot than to opting out of training.

The third group is on-demand fetchers, which retrieve a page at the moment a user asks about it: ChatGPT-User, Claude-User, Perplexity-User. A hit from one of these usually means a real human, right now, is reading your content through an AI assistant. It is the closest thing this ecosystem has to a referral.

One entry deserves a special note because it confuses everyone: Google-Extended is not a crawler and never appears in your logs. It is a robots.txt token that tells Google’s regular crawlers whether your content may be used for AI training. Ordinary Googlebot does the fetching either way. If a log-analysis guide tells you to look for Google-Extended hits, that guide is guessing.

Pulling Them Out of the Log

On a standard access log, a first pass is one command:

grep -iE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|perplexitybot|ccbot|bytespider|amazonbot|meta-external" access.log | awk '{print $1}' | sort | uniq -c | sort -rn

That gives you hit counts per IP for the whole AI cohort. Two refinements make it genuinely useful. Split the counts by user agent rather than lumping them, so you can see whether you are dealing with training collection or live user fetches:

grep -ioE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|perplexitybot|ccbot|bytespider" access.log | sort | uniq -c | sort -rn

And look at what they fetch, not just how much. A training crawler that pulls your entire archive once a month is a different story from an on-demand fetcher hitting the same product page forty times a day. The second one is telling you which of your pages AI users actually ask about, which is free market research if you bother to read it.

The Impostor Problem

Here is the part most posts skip: a user-agent string is a claim, not a credential. Anyone can send GPTBot in a header. Scrapers routinely impersonate well-known AI bots precisely because many sites now treat those names gently. Before you conclude anything from your counts, and long before you make blocking decisions based on them, verify.

The clean method is two-sided DNS verification. Take the IP, do a reverse lookup, and check that the resulting hostname belongs to the operator’s domain; then resolve that hostname forward and confirm it returns the same IP:

host 203.0.113.42          # reverse: should end in something like openai.com
host <returned-hostname>   # forward: should resolve back to 203.0.113.42

The major operators also publish their crawler IP ranges as machine-readable lists (OpenAI, Anthropic, and Perplexity all do), so you can check a suspect IP against the published ranges directly. In practice a meaningful share of self-declared AI-bot traffic fails these checks. Treat verified and unverified hits as two separate datasets: one tells you about AI companies, the other tells you about scrapers wearing their jackets.

What to Actually Do With This

Once you know who is really visiting, the decisions become ordinary policy decisions rather than panic. If you do not want to feed training corpora, disallow the training group in robots.txt; the syntax and the trade-offs are covered in our complete guide to robots.txt and AI crawlers. If you care about AI search visibility, be careful not to block the index crawlers and the on-demand fetchers while aiming at the training bots; that specific friendly-fire mistake is common and quietly removes you from AI answers. And if a heavy crawler is straining your server, remember that well-behaved bots respect Crawl-delay and robots rules after verification, while impostors do not, which is exactly why the verification step comes first: you rate-limit or block impostors at the firewall, not in robots.txt, because robots.txt is a note that only honest actors read.

The last habit worth building is comparing the AI cohort’s behavior against your own crawl of your site. The bots see the site you actually serve, not the site you think you serve. Running a full crawl shows you what any crawler, search or AI, encounters: the broken links, the redirect chains, the accidentally blocked sections. If GPTBot is spending its visits on parameter-soup URLs you forgot existed, the fix is not an AI policy, it is ordinary technical hygiene, and a technical SEO audit will surface it.

The Bottom Line

Your server logs are the only honest record of the AI era’s most consequential visitors, and reading them takes one grep and one verification habit. Sort the visitors into training, index, and on-demand groups, verify before you trust any name, and make blocking decisions per group rather than in one sweep. The bots are not going away; the sites that handle them well will be the ones that know exactly who is at the door.