AI Crawler Log Analysis: Verify GPTBot, ClaudeBot and PerplexityBot, Then Measure What They Actually Fetch

Every SEO now wants to know what the AI crawlers are doing on their site, and almost every answer to that question is a single grep for GPTBot followed by a screenshot. That is not analysis. A user-agent string is a claim, not an identity, and a count of requests tells you nothing about whether the model behind the crawler can actually read your pages. AI crawler log analysis done properly is a small engineering pipeline: extract the candidate requests, verify each one against the operator’s published IP ranges, then compute a handful of per-bot metrics that map to decisions you can act on.

This guide is that pipeline. It assumes you already have access logs and have read the general log file analysis guide, which covers parsing, crawl-budget waste and the basic AI crawler grep. Here we go further: verification code you can run, the metrics that separate training crawls from live answer fetches, and how to read the output without fooling yourself.

Key takeaways

• AI crawler user agents are self-reported, so every request has to be verified against the operator’s published IP list before it counts; OpenAI, Anthropic and Perplexity all publish one.
• There is no Google-Extended request to find in a log: Google documents that it has no separate user agent and works only as a robots.txt token.
• The metric that predicts AI visibility is not GPTBot volume. It is the count of verified user-initiated fetches (ChatGPT-User, Claude-User, Perplexity-User) on content URLs returning full HTML.
• Sorting a bot’s 200 responses by byte size is the fastest way to find pages that return an empty JavaScript shell to crawlers that do not render.
• Block decisions belong in a WAF rule that combines user agent and IP range, not in an IP-only block, because IP blocks stop the operator reading your robots.txt.

What AI crawler log analysis answers that nothing else can

Three tools claim to tell you about AI crawlers, and only one of them is evidence. Analytics platforms never see a crawler, because crawlers do not run the tag. Visibility tools tell you whether an answer engine cited you, not whether its crawler could fetch you. The access log is the only record of the request the crawler made, the response your server sent and the size of what went back.

That makes AI crawler log analysis the answer to four questions the other tools cannot touch. Which of your URLs have the training crawlers seen, and how recently. Whether the live fetchers that power answers, the user-initiated agents, are hitting your content pages or bouncing off redirects. Whether what they receive is the page or an empty shell. And whether the traffic claiming to be an AI crawler is real, which matters when you are about to block it or, worse, to build a strategy around it.

The reason it needs engineering rather than a grep is verification. Scrapers spoof crawler user agents routinely. OpenAI’s crawler documentation publishes example user-agent strings and warns that the version number may change, and it publishes an IP list for each bot for exactly this reason: the string is a label, the IP range is the identity.

The bots, and what their user agents actually look like

The AI operators split their traffic by purpose, and the split is what makes the metrics meaningful. Each operator runs a training crawler, a search-index crawler and a user-initiated fetcher, and they behave differently in your logs.

Operator	Token	Purpose	Obeys robots.txt	IP list
OpenAI	GPTBot	Training foundation models	Yes	openai.com/gptbot.json
OpenAI	OAI-SearchBot	ChatGPT search index	Yes	openai.com/searchbot.json
OpenAI	ChatGPT-User	Fetch on a user’s request	May not apply	openai.com/chatgpt-user.json
Anthropic	ClaudeBot	Training	Yes, plus Crawl-delay	claude.com/crawling/bots.json
Anthropic	Claude-SearchBot	Search indexing	Yes	same list
Anthropic	Claude-User	Fetch on a user’s request	Documented as respecting robots.txt	same list
Perplexity	PerplexityBot	Search index, not training	Yes	perplexity.com/perplexitybot.json
Perplexity	Perplexity-User	Fetch on a user’s request	Generally ignores	perplexity.com/perplexity-user.json
Google	Google-Extended	Gemini training and grounding control	Token only	none, no user agent

Two details from the operators’ own documentation change how you grep. First, OpenAI says that when its bots fetch robots.txt they may add a literal robots.txt marker to the user-agent string, so a request like GPTBot/1.4; robots.txt is a robots fetch, not a content fetch, and should be counted separately. Second, Google’s crawler list states that Google-Extended has no separate HTTP user agent, crawling is done with existing Google user agent strings and the token is used only in a control capacity in robots.txt. If a dashboard shows you Google-Extended requests, the dashboard is wrong.

The split between crawler and user fetcher also decides what robots.txt can do. OpenAI documents that ChatGPT-User is not used for automatic crawling and that because its actions are initiated by a user, robots.txt rules may not apply. Perplexity is blunter: since a user requested the fetch, Perplexity-User generally ignores robots.txt. So a Disallow for the training crawlers will change your training-crawler numbers and leave your user-fetch numbers alone, which is exactly why the two have to be reported separately. The policy side of that decision is covered in robots.txt for developers.

Step 1: extract candidate requests from the access log

Start with a superset. Pull every line whose user-agent field contains any AI crawler token, and keep the fields you will need downstream: timestamp, IP, method, path, status, bytes and the full user agent. The example below assumes the Nginx combined format; adjust the field numbers for Apache or a CDN export.

# Candidate AI-crawler requests, tab-separated for downstream tools
grep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' access.log \
  | awk -F'"' '{
      split($1, a, " ");                 # a[1]=ip, a[4]=[timestamp
      split($2, r, " ");                 # r[1]=method, r[2]=path
      split($3, s, " ");                 # s[1]=status, s[2]=bytes
      print a[1] "\t" a[4] "\t" r[1] "\t" r[2] "\t" s[1] "\t" s[2] "\t" $6
    }' > ai_candidates.tsv

Call the file candidates, and mean it. Nothing in it has been verified yet. If your logs sit behind a CDN, make sure the IP field is the client IP the CDN forwarded, not the CDN edge, or every verification will fail.

Add a column for the token and one for whether the request was a robots.txt fetch. Both are cheap to derive and both matter later:

awk -F'\t' 'BEGIN{OFS="\t"} {
  ua=$7; tok="other";
  if (ua ~ /GPTBot/) tok="GPTBot";
  else if (ua ~ /OAI-SearchBot/) tok="OAI-SearchBot";
  else if (ua ~ /ChatGPT-User/) tok="ChatGPT-User";
  else if (ua ~ /ClaudeBot/) tok="ClaudeBot";
  else if (ua ~ /Claude-SearchBot/) tok="Claude-SearchBot";
  else if (ua ~ /Claude-User/) tok="Claude-User";
  else if (ua ~ /PerplexityBot/) tok="PerplexityBot";
  else if (ua ~ /Perplexity-User/) tok="Perplexity-User";
  robots = ($4 == "/robots.txt" || ua ~ /robots\.txt/) ? 1 : 0;
  print $0, tok, robots
}' ai_candidates.tsv > ai_candidates_tagged.tsv

Step 2: verify every request against the published IP lists

This is the step most write-ups skip, and it is the one that makes the rest trustworthy. Each operator publishes its source addresses as JSON. The GPTBot list is a small object with a creationTime and a prefixes array of ipv4Prefix CIDR blocks, and the OAI-SearchBot and ChatGPT-User lists use the same shape. Anthropic publishes a single list for all three of its bots and says that a crawler with a source IP on it is coming from Anthropic. Perplexity publishes one list per bot and recommends that WAF rules combine the user-agent string with the IP ranges.

The verifier below fetches the lists, builds one network set per token and stamps every candidate line as verified, spoofed or unknown. It uses only the Python standard library.

import csv, ipaddress, json, urllib.request

LISTS = {
    "GPTBot":           "https://openai.com/gptbot.json",
    "OAI-SearchBot":    "https://openai.com/searchbot.json",
    "ChatGPT-User":     "https://openai.com/chatgpt-user.json",
    "ClaudeBot":        "https://claude.com/crawling/bots.json",
    "Claude-SearchBot": "https://claude.com/crawling/bots.json",
    "Claude-User":      "https://claude.com/crawling/bots.json",
    "PerplexityBot":    "https://www.perplexity.com/perplexitybot.json",
    "Perplexity-User":  "https://www.perplexity.com/perplexity-user.json",
}

def load_networks(url):
    data = json.load(urllib.request.urlopen(url, timeout=20))
    nets = []
    for p in data.get("prefixes", []):
        cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
        if cidr:
            nets.append(ipaddress.ip_network(cidr))
    return nets

networks = {tok: load_networks(url) for tok, url in LISTS.items()}

def verify(ip, token):
    try:
        addr = ipaddress.ip_address(ip)
    except ValueError:
        return "unknown"
    nets = networks.get(token)
    if not nets:
        return "unknown"
    return "verified" if any(addr in n for n in nets) else "spoofed"

with open("ai_candidates_tagged.tsv") as f, open("ai_verified.tsv", "w", newline="") as out:
    reader = csv.reader(f, delimiter="\t")
    writer = csv.writer(out, delimiter="\t")
    for row in reader:
        ip, token = row[0], row[7]
        writer.writerow(row + [verify(ip, token)])

Three rules keep this honest. Cache the lists with their creationTime and refresh them daily, because the ranges change and a stale list turns real traffic into false spoofs. Treat unknown as a category to report, not to hide; it is where IPv6 and new bots land. And never verify Anthropic’s bots by reverse DNS alone. Anthropic’s list is the documented method, and its documentation also warns that blocking by IP may not persistently guarantee an opt-out because it impedes the operator’s ability to read your robots.txt, which is a hint about how the ranges are operated.

The first number to report is the spoof rate per token, because every downstream metric is meaningless until the rows from outside the published ranges are excluded.

Step 3: the per-bot metrics that map to decisions

With a verified file, the interesting work is aggregation. Load the TSV into DuckDB, SQLite or a spreadsheet and compute the following. The SQL is DuckDB syntax; the column order matches the TSV above.

CREATE TABLE ai AS
SELECT column0 AS ip, column1 AS ts, column2 AS method, column3 AS path,
       column4::INT AS status, column5::BIGINT AS bytes, column6 AS ua,
       column7 AS token, column8::INT AS robots, column9 AS verdict
FROM read_csv('ai_verified.tsv', delim='\t', header=false);

-- 1. Volume, spoof rate and robots.txt share per token
SELECT token,
       count(*)                                          AS requests,
       round(100.0 * sum(verdict='spoofed') / count(*), 1) AS spoof_pct,
       round(100.0 * sum(robots) / count(*), 1)          AS robots_pct
FROM ai GROUP BY token ORDER BY requests DESC;

Everything after this filters to verdict = 'verified' AND robots = 0.

Unique content URLs per token, and coverage against the sitemap. A training crawler that fetched 40,000 requests but 300 unique URLs is recrawling the same pages. Join the distinct paths against your sitemap URLs and report the percentage of canonical pages each bot has fetched at least once in the window. This is the AI equivalent of the orphan-page check in the general guide.

Status mix per token. The share of 200, 301/302, 404 and 5xx responses. Redirect chains cost a user fetcher time it may not spend, and a 404 rate on URLs that exist means the bot is following stale links from somewhere, usually an old sitemap or a citation to a URL you moved without redirecting.

Small-body 200s on content URLs. This is the one that catches client-rendered pages. None of these crawlers run JavaScript, so a page that hydrates in the browser returns an almost empty shell to them with a healthy status code. Compute the median bytes per token for content paths and list the URLs below a threshold you set from your own template sizes.

-- 2. Shells: successful content fetches with suspiciously small bodies
SELECT token, path, bytes
FROM ai
WHERE verdict='verified' AND robots=0 AND status=200
  AND path NOT LIKE '%.js' AND path NOT LIKE '%.css' AND path NOT LIKE '%.xml'
  AND bytes < 4000
ORDER BY bytes ASC LIMIT 50;

User-fetch to crawl ratio. Count verified requests from the user-initiated agents (ChatGPT-User, Claude-User, Perplexity-User) separately from the crawlers, and report the ratio per week. This is the metric that actually tracks visibility. A training crawl says the operator wants your content in a corpus. A user fetch says an answer engine reached for your page while composing a reply to a real person, and the URL it fetched is the URL it was considering citing. Group user fetches by path, and you have a first-party list of the pages that answer engines are actively using.

Recrawl interval. For each token and path, the time between successive fetches. Short intervals on pages that have not changed are wasted server time; long intervals on pages you update weekly mean the model’s view of them is stale.

Time of day and burstiness. Requests per minute at the 99th percentile per token. This is what you need before deciding whether to set Crawl-delay for ClaudeBot, which Anthropic documents as a supported non-standard extension, or whether the bursts are actually the spoofed rows you already excluded.

Step 4: turn the numbers into decisions

The output of AI crawler log analysis is a short list of actions, and each metric above points at one.

• Spoof rate is high. Add a WAF rule per token that allows the user agent only from the published ranges and challenges or blocks it otherwise. This is the configuration Perplexity’s documentation describes for Cloudflare and AWS WAF, and it works for every operator that publishes a list.
• Shells are appearing. Fix rendering for the affected templates. Server-render or pre-render the content, or at minimum ensure the HTML response contains the headline and body text before any JavaScript runs. The check is the same as for Googlebot, and JavaScript SEO covers the options.
• User fetches concentrate on a few pages. Those pages are your AI answer surface. Keep them fast, keep their URLs stable, and make sure their structure gives an answer engine something quotable, which is the subject of how to get cited by ChatGPT.
• Training crawl volume is high and user fetches are low. Nothing about your visibility follows from the crawl volume. If the crawl is costing you server time, this is where Crawl-delay for ClaudeBot or a Disallow for GPTBot is a defensible decision, because OpenAI documents the training and search tokens as independent switches and Google documents that Google-Extended does not affect Search inclusion or ranking.
• Redirect and 404 rates are high. Trace where the bot got the URLs. User fetchers usually follow links that appear in answers, so a 404 there is a citation to a dead page.

One rule ties all of this together: never block by IP alone. Anthropic’s own guidance is explicit that IP blocking may stop its crawler reading your robots.txt, which is the mechanism you are relying on to express your preference. Express the policy in robots.txt, enforce identity at the WAF, and let the log tell you whether both are working.

What the logs cannot tell you

AI crawler log analysis is bounded by what a request carries. A fetch by ChatGPT-User tells you a page was retrieved during a conversation. It does not tell you whether the answer quoted you, how you were described, or whether the user clicked through. The first two are what visibility tools and a manual prompt matrix measure, covered in how to check whether your business appears in ChatGPT and Perplexity. The click is what your analytics sees, as far as consent and referrers allow, which is the subject of tracking AI referral traffic in GA4.

There is also a category the logs cannot separate at all. Google’s AI Overviews and AI Mode are built from Googlebot’s ordinary crawl, and Google-Extended is a token, not a crawler, so there is no log line that says a fetch was for an AI feature rather than for the classic index. For Google, the log confirms fetchability and nothing more.

Run the pipeline monthly, keep the per-token table as a time series, and the trend in verified user fetches on content URLs becomes the most honest AI visibility metric you own, because it comes from your server rather than from a vendor’s sample.

Frequently asked questions

How do I verify that a GPTBot request is really from OpenAI?

Match the request’s source IP against the CIDR prefixes in the JSON list OpenAI publishes at openai.com/gptbot.json, and use the separate searchbot.json and chatgpt-user.json lists for OAI-SearchBot and ChatGPT-User. A request whose user agent says GPTBot but whose IP falls outside those ranges is spoofed and should be excluded from every metric. Refresh the lists daily, because the prefixes change.

Why does Google-Extended never appear in my server logs?

Because it is not a crawler. Google documents that Google-Extended has no separate HTTP user agent, that crawling is done with existing Google user agent strings, and that the token only works as a control in robots.txt for Gemini training and grounding. The fetches you can see are from Googlebot and GoogleOther, and they serve both classic Search and Google’s AI features.

Which AI crawler metric best predicts whether I get cited?

Verified fetches from the user-initiated agents, ChatGPT-User, Claude-User and Perplexity-User, on content URLs that return full HTML. Those fetches happen while an answer is being composed, so the URLs they hit are the pages the engines are actively considering. Training-crawler volume from GPTBot or ClaudeBot says nothing about citation.

Can robots.txt stop ChatGPT-User or Perplexity-User fetching my pages?

Not reliably. OpenAI states that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply, and Perplexity states that Perplexity-User generally ignores robots.txt for the same reason. If you need to stop user-initiated fetches, the control is a WAF rule keyed on the user agent and the published IP ranges, not a Disallow line.

How do I find pages that return an empty shell to AI crawlers?

Filter a bot’s verified 200 responses to content paths and sort by response size, smallest first. Pages far below your template’s normal size are returning HTML without the rendered content, which happens when the page depends on client-side JavaScript. Confirm by requesting the page with the bot’s user agent and checking whether the headline and body are present in the raw HTML.