Log File Analysis for SEO: Reading What Bots Actually Fetched

What is log file analysis for SEO?

Log file analysis for SEO is the practice of reading a server’s raw access logs to see exactly which URLs search-engine and AI crawlers requested, what status code each request returned, and how much crawl activity is being spent where. Unlike a crawl simulation or Search Console’s sampled reports, the access log is a complete, first-hand record of every fetch a bot actually made — which makes it the only place you can confirm what Googlebot, GPTBot, ClaudeBot, and PerplexityBot really saw.

I do both sides of this — I write the code and I do the SEO — and log analysis is the one technical-SEO task that lives entirely on the server, in a file most marketers never open. Everything else in an audit is an inference about crawler behaviour. The log is the behaviour itself. This guide is the implementation-first version: real access-log lines, real grep/awk you can paste into a terminal, and a dedicated section on the part almost nobody covers yet — filtering logs by AI-crawler user-agent to verify the models can actually reach your content.

TL;DR — Key takeaways

• The access log is the only ground truth for crawl behaviour. A Screaming Frog crawl tells you what a bot could fetch; Search Console’s Crawl Stats report is sampled and aggregated. The raw log is every request, with the real status code and byte count. When those three sources disagree, the log wins.

• You do not need a tool to start. The combined log format is space-delimited, and grep, awk, sort, and uniq will answer most questions: hits per URL, per status code, per user-agent, per bot. GoAccess and Screaming Frog’s Log File Analyser are conveniences on top of that, not prerequisites.

• Crawl-budget waste is visible in one command. Bots hammering faceted and parameter URLs, repeatedly hitting 404s, or chasing redirect chains are all one awk filter away. On large sites this is where wasted crawl gets reclaimed for the pages that matter.

• Never trust a user-agent string on its own. Anyone can send User-Agent: Googlebot. Verify with reverse-DNS forward confirmation, or match the source IP against the crawler operator’s published IP ranges — Google, OpenAI, Perplexity, and Anthropic all publish them now.

• AI crawlers leave the same footprints, and they are the fresh reason to read logs in 2026. GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), and PerplexityBot all fetch raw HTML and never run JavaScript. Filtering the log by their user-agents tells you which pages the models retrieve — and a suspiciously tiny response to GPTBot on a content URL usually means it received a JavaScript shell, not your article.

• “Google-Extended” will never appear in your logs, and that trips people up. It is a robots.txt control token that governs whether Google may use already-crawled content for Gemini training and grounding — Google’s documentation is explicit that it has no separate request user-agent. Google’s actual fetches show up as Googlebot and GoogleOther.

The access log is the only record of what bots actually fetched

Every request to your server writes a line to an access log. On Apache and Nginx the default is the combined log format, and once you can read one line you can read a million of them. Here is a single request from Googlebot:

66.249.66.1 - - [13/Jul/2026:08:14:59 +0000] "GET /insights/log-file-analysis-seo/ HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The fields, in order:

• 66.249.66.1 — the client IP. For a bot, this is what you verify against published ranges.
• - - — the identd and HTTP-auth user, almost always empty.
• [13/Jul/2026:08:14:59 +0000] — the timestamp.
• "GET /insights/log-file-analysis-seo/ HTTP/1.1" — the request line: method, URL path, protocol.
• 200 — the HTTP status code.
• 18342 — the response size in bytes. This one matters more than people expect for AI-crawler work.
• "-" — the referer.
• "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" — the user-agent string.

That is the whole record. The reason it beats every other data source is that it is neither sampled nor simulated. Search Console’s Crawl Stats report is a useful summary, but it is aggregated and it only covers Google. A crawler like Screaming Frog reproduces what a bot would see, not what Googlebot’s renderer with its own timeouts and crawl-budget logic actually did — a gap I covered in technical SEO for headless architecture. The access log has no such gap. It is the request, as it happened.

Parsing access logs with grep and awk

With awk splitting on whitespace, the request URL is field $7, the status code is $9, the byte size is $10, and the IP is $1. The user-agent contains spaces, so the cleanest way to isolate it is to split on the double-quote character instead, where it lands in field $6. That is the entire mental model. Everything below is a variation on it.

What are bots requesting most?

# Top 20 most-requested URLs across all clients
awk '{print $7}' access.log | sort | uniq -c | sort -rn | head -20

What status codes is the server returning?

# Response-code distribution — a spike in 404s or 301s is the first red flag
awk '{print $9}' access.log | sort | uniq -c | sort -rn

Which user-agents are hitting the site?

# Split on the quote char so the full user-agent is one field
awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20

What is one specific bot fetching?

# The pages Googlebot spends its crawl on, most-hit first
grep 'Googlebot' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

# A raw count of GPTBot requests in the file
grep -c 'GPTBot' access.log

From here every useful question is a combination of a grep (pick the bot) and an awk filter (pick the condition). That composability is why the command line is still the fastest way in, even when a GUI tool is sitting right there.

Finding crawl-budget waste

Crawl budget is only a real constraint on large or rapidly-changing sites — a small static blog is crawled comfortably within whatever budget Google assigns it. But when it does bite, logs are where you see it being wasted, because waste has a signature: bots making large numbers of requests to URLs that should never have earned them.

Faceted and parameter URLs. Filter, faceting, and session parameters can generate a near-infinite URL space, and crawlers will happily wander into it. Find the parameterised URLs bots are burning requests on:

# Bot requests to URLs containing a query string, most-hit first
grep -i 'bot' access.log | awk '$7 ~ /\?/ {print $7}' | sort | uniq -c | sort -rn | head -20

If a handful of ?sort=, ?filter=, or ?sessionid= patterns dominate that list, you have found crawl waste that robots.txt rules or rel="canonical" consolidation should be reclaiming.

404s that bots keep hitting. A page returning 404 to real users is one problem; a bot requesting a dead URL hundreds of times is a crawl-budget problem on top of it, usually caused by a stale internal link or an old URL still in the sitemap:

# Dead URLs that crawlers are still requesting
grep -i 'bot' access.log | awk '$9 == 404 {print $7}' | sort | uniq -c | sort -rn | head -20

Redirect chains. Every redirect hop is a wasted round-trip, and a chain multiplies it. If a bot is repeatedly receiving 301s on a set of URLs, those are candidates to collapse into a single direct redirect to the final destination — the exact failure mode that spikes during a domain move, which is why I treat it as a checklist item in the site migration guide:

# URLs Googlebot is being redirected on (301/302), most frequent first
grep 'Googlebot' access.log | awk '$9 ~ /^30[12]$/ {print $7}' | sort | uniq -c | sort -rn | head -20

The pattern across all three is the same: a bot spending requests on URLs that will never rank or be cited is spending requests it could have spent on the ones that will.

Finding orphan pages: logs vs. sitemap

An orphan page is one that exists but receives no internal links, so link-following crawlers rarely reach it. Logs let you detect two related conditions by comparing the set of URLs bots actually crawled against the set of URLs in your sitemap — in both directions.

First, build the two sets. The URLs Googlebot successfully fetched:

# Unique paths Googlebot fetched with a 200, sorted for comparison
grep 'Googlebot' access.log | awk '$9 == 200 {print $7}' | sort -u > crawled.txt

And the paths declared in your sitemap (stripping the origin so the two lists are comparable):

# Extract <loc> URLs and reduce them to paths
curl -s https://example.com/sitemap.xml \
  | grep -oE 'https?://[^<]+' \
  | sed -E 's#https?://[^/]+##' \
  | sort -u > sitemap-paths.txt

Now comm does the set arithmetic:

# In the sitemap but never crawled — discovery or priority problem
comm -23 sitemap-paths.txt crawled.txt

# Crawled but not in the sitemap — orphaned content, or junk earning crawl it shouldn't
comm -13 sitemap-paths.txt crawled.txt

The first list is pages you have told Google to index that it is not bothering to fetch — often because nothing internal links to them, or because they sit too deep in the architecture. The second list is URLs bots reach that you never declared: sometimes genuinely orphaned content worth surfacing, sometimes parameter noise or leftover URLs that should be pruned. Either way, the log turned an abstract “do we have orphan pages?” into a concrete, reviewable list.

Verifying bot identity properly — don’t trust the user-agent

The user-agent string is self-reported and trivially spoofed. Scrapers routinely send Googlebot to slip past rules, so before you act on any “Googlebot is doing X” finding, confirm the request really came from Google. There are two legitimate methods.

Reverse-DNS forward confirmation. This is Google’s own documented verification. Take the IP from the log line, reverse-look it up, confirm the hostname belongs to the operator, then forward-confirm that the hostname resolves back to the same IP:

# 1. Reverse lookup — what hostname claims this IP?
host 66.249.66.1
#   → ...domain name pointer crawl-66-249-66-1.googlebot.com

# 2. Forward-confirm — does that hostname resolve back to the same IP?
host crawl-66-249-66-1.googlebot.com
#   → crawl-66-249-66-1.googlebot.com has address 66.249.66.1

A legitimate Googlebot IP resolves to googlebot.com or google.com and round-trips cleanly. If the reverse lookup points somewhere else — or doesn’t round-trip — the request is not Google, regardless of what the user-agent claims. Bingbot verifies the same way against search.msn.com.

Published IP ranges. The modern alternative is to match the source IP against the operator’s official list. Google publishes its crawler ranges as JSON (common-crawlers.json and related files linked from its crawler documentation); the AI operators publish theirs too, covered in the next section. Matching an IP against a published range is faster than a DNS round-trip when you are validating in bulk, and it is the approach the crawler operators themselves recommend for firewall and WAF rules.

Auditing AI crawlers in your logs

This is the part that makes log analysis a 2026 skill rather than a 2015 one. The models behind ChatGPT, Claude, and Perplexity reach your site through named crawlers, and those crawlers write the same log lines as Googlebot. Filtering the log by their user-agents is the only way to confirm — from your own server — which of your pages the AI engines are actually retrieving. Verified against each operator’s own documentation, the tokens to grep for are:

• OpenAI — GPTBot (trains foundation models), OAI-SearchBot (powers ChatGPT search results), and ChatGPT-User (user-initiated fetches inside ChatGPT). Each has a published IP list at openai.com/gptbot.json, openai.com/searchbot.json, and openai.com/chatgpt-user.json.
• Anthropic — ClaudeBot (training), Claude-SearchBot (search indexing), and Claude-User (user-directed retrieval). Anthropic publishes a verification IP list at claude.com/crawling/bots.json.
• Perplexity — PerplexityBot (indexing) and Perplexity-User (browsing on behalf of a live user), with IP lists at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json.
• Google — there is no AI-specific user-agent to grep for. Google-Extended is a robots.txt token that governs whether Google may use content for Gemini training and grounding; the fetches themselves come from Googlebot and the general-purpose GoogleOther.

One command gives you the whole AI-crawler picture:

# How often each AI crawler hit the site
grep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' access.log \
  | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

What “good” looks like. A healthy AI-crawler footprint is the bots fetching your real content URLs — articles, docs, product pages — and receiving 200 responses with byte counts consistent with a full page. What you are hunting for is the opposite: AI bots getting 404s on content that exists, being redirected repeatedly, or — the subtle one — receiving a 200 with a suspiciously small response body. Because none of these crawlers execute JavaScript, a client-rendered page hands them an almost-empty HTML shell while returning a perfectly healthy status code. Sort GPTBot’s successful fetches by size, smallest first, and the shells float to the top:

# GPTBot's 200 responses, smallest payload first — tiny sizes on content URLs = JS shells
grep 'GPTBot' access.log | awk '$9 == 200 {print $10, $7}' | sort -n | head -20

If a content URL is returning a few hundred bytes to GPTBot, confirm what the bot receives by requesting the page as it would:

# Ask for the page as GPTBot; if your H1 and body aren't here, neither engine can cite them
curl -sA "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" https://example.com/page | grep -i '<h1'

That curl check is the same crawlable-vs-rendered test I walk through for engineers in SEO for engineers — the log just tells you which pages to run it on. This is where log analysis connects directly to AI visibility: a page an AI crawler cannot fetch as real HTML is a page the model cannot retrieve, and a page the model cannot retrieve is one it cannot cite. Being fetchable is the precondition for everything downstream.

38%

of Google AI Overview citations come from pages already ranking in Google’s top 10

Source: Ahrefs — down from ~76% in July 2025 as Google shifted toward query fan-out

67%

higher AI citation eligibility for content with valid schema markup

Source: ConvertMate — analysis of 80M+ AI citations

The tactical follow-through — turning a page an AI crawler can reach into one it actually quotes — is the subject of how to get cited by ChatGPT. And once the bots are fetching cleanly, the referral side of the same story shows up in analytics, which is why I pair log analysis with tracking AI referral traffic: the log proves the crawler came in, the referral data shows the humans it sent back.

Tools beyond the command line

The CLI answers most questions, but two tools are worth knowing for larger or recurring work. I am deliberately describing only what I can state accurately — verify current pricing and limits with each vendor before you rely on them.

• GoAccess — a free, open-source log analyser that parses the common and combined log formats and renders a live dashboard, either in the terminal or as a self-contained HTML report. It breaks traffic down by URL, status code, referrer, and user-agent, which covers the aggregate view without any scripting. Good for a fast visual pass over a large file.
• Screaming Frog Log File Analyser — a dedicated desktop application (distinct from the SEO Spider crawler) that imports raw access logs and aggregates them by URL, response code, and user-agent, with built-in verification of the major search-engine bots. It offers a free version alongside a paid licence for larger logs. Useful when you want to join log data against a site crawl in one interface.

Neither replaces the grep/awk approach so much as sits on top of it. When I need a repeatable answer to a specific question — “which content URLs is GPTBot getting shells for?” — the command line is faster. When I want a browsable overview to hand to someone else, GoAccess or the Log File Analyser earns its place.

Where this fits in the bigger picture

Log analysis is the verification layer under the rest of technical SEO. The technical SEO audit methodology tells you what to check; the log tells you what the crawler actually did about it. When an audit finds JavaScript-rendered content, the log confirms whether AI crawlers are receiving shells. When a site migration collapses a URL structure, the log is where you watch redirect chains resolve and 404s disappear. When a headless architecture is in question, the log settles what each template really serves to a non-JavaScript bot.

If you would rather have this instrumented and interpreted end-to-end than run it yourself, that is part of what I do: GEO and technical SEO consulting covers the crawl, rendering, and AI-citation-readiness layers as one scope, and full-stack development covers building the logging, parsing, and server-render fixes into the codebase directly. Either way, the principle holds: stop inferring crawler behaviour from tools that simulate it, and read the record of what actually happened.

FAQ

What is log file analysis in SEO?

Log file analysis in SEO is reading a web server’s raw access logs to see exactly which URLs search-engine and AI crawlers requested, what HTTP status each request returned, and how crawl activity is distributed. Because the log is a complete, first-hand record rather than a sample or a simulation, it is the authoritative source for what Googlebot, GPTBot, ClaudeBot, and PerplexityBot actually fetched — used to find crawl-budget waste, orphan pages, and rendering problems that other tools only approximate.

What log format do I need, and where do I get the file?

Most servers default to the combined log format, a space-delimited line containing the client IP, timestamp, request line (method, URL, protocol), status code, response size, referer, and user-agent. On Apache and Nginx it is enabled out of the box. You retrieve the file from the server itself (commonly under /var/log/), from your hosting control panel, or from your CDN’s log-export feature if traffic terminates at a CDN before reaching origin.

How do I verify a request is really from Googlebot and not a spoofer?

Never trust the user-agent string alone — it is self-reported and easily faked. Use reverse-DNS forward confirmation: reverse-look up the request’s IP with host <ip>, confirm the hostname belongs to googlebot.com or google.com, then forward-confirm that the hostname resolves back to the same IP. Alternatively, match the source IP against Google’s published crawler IP ranges. OpenAI, Perplexity, and Anthropic publish equivalent IP lists for their crawlers.

Which AI crawler user-agents should I look for in my logs?

The verified tokens are GPTBot, OAI-SearchBot, and ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot, and Claude-User (Anthropic); and PerplexityBot and Perplexity-User (Perplexity). Google has no AI-specific user-agent — Google-Extended is a robots.txt control token with no separate request user-agent, so Google’s fetches appear as Googlebot and GoogleOther. Grep the log for those strings and split on the quote character to isolate the user-agent field.

Why do AI crawlers matter in log analysis specifically?

Because GPTBot, ClaudeBot, PerplexityBot, and their siblings fetch raw HTML and never execute JavaScript, the log is where you confirm they can actually reach your content. A client-rendered page returns a healthy 200 status but an almost-empty HTML shell to these bots, so their requests look fine in a status-code summary while the models receive nothing to cite. Sorting an AI crawler’s successful fetches by response size surfaces those shells — tiny byte counts on real content URLs are the tell.

Do I need a paid tool to analyse log files?

No. grep, awk, sort, and uniq answer most questions — hits per URL, per status code, per user-agent, and per bot — directly against the raw file. GoAccess is a free, open-source dashboard for a browsable overview, and Screaming Frog’s Log File Analyser is a paid desktop option (with a limited free version) for joining log data against a site crawl. Tools add convenience; they do not add information the log doesn’t already contain.