Log file analysis for SEO is the practice of reading your server’s access logs to see exactly which URLs crawlers requested, how often, and what each request returned. It is the only data source that shows what Googlebot and other bots actually did on your site, not what a crawler simulation guesses they might do. If you want the truth about how search engines experience your site, the answer is sitting in your logs.
Every other technical SEO tool is a model. A crawler like Screaming Frog imitates a bot. Search Console samples and aggregates. Your access logs record reality: one line per request, timestamped, with the status code the server actually sent back. Across the projects I have worked on, the most expensive crawl problems were invisible in every tool except the raw logs.
This guide covers how to get the logs, what to look for, how to verify bots in a world full of fake ones, and the framework I use to turn a wall of log lines into ranking gains.
What a Server Log Actually Records
A server log is a plain-text file where your web server writes one line for every request it answers. The de-facto standard for SEO work is the Apache/Nginx combined log format, and once you can read a single line you can read the whole file.
A typical line looks like this:
66.249.66.1 - - [30/Jun/2026:08:14:22 +0000] "GET /services/seo-audit/ HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Read left to right, that single line tells you the client IP, the timestamp, the request method and URL, the HTTP status code, the bytes sent, the referrer, and the user-agent. Multiply that by a few million lines and you have a complete, unfiltered record of every crawler visit your site received.
The fields that matter most for SEO are the IP (for bot verification), the requested URL, the status code, and the user-agent. Everything useful in log file analysis comes from grouping and counting those four fields.
Where your logs live
On a traditional Apache or Nginx stack the access logs sit in /var/log/. If you run behind Cloudflare, Fastly, or another CDN, the origin only sees a fraction of traffic, so you need the edge logs exported through the provider’s dashboard or a logpush job. Managed platforms like Vercel, Netlify, or a typical WordPress host vary — some expose logs in the dashboard, others require a support request or a log-drain integration.
The rule is simple: get raw, unsampled, unfiltered logs covering every subdomain, for at least 30 days. Anything pre-aggregated has already thrown away the detail you need.
Why Log File Analysis Beats Every Crawl Simulation
Search Console’s Crawl Stats report is a good starting point, but it samples and only shows verified Googlebot. It will never show you Bing, GPTBot, ClaudeBot, or the fake bots hammering your origin. Raw logs show all of them. This is the gap that crawl budget optimization lives in: Google maintains an index of over 400 billion documents, and on a large site Googlebot will only ever sample a slice of your URLs per day. Logs tell you which slice.
In one audit, a client’s most important category pages were being crawled once a month while Googlebot burned thousands of daily requests on parameter URLs from a broken faceted-navigation setup. No crawl tool flagged it, because the parameter URLs were technically valid. Only the logs showed the misallocation — and fixing it cut time-to-index for new products from weeks to days.
That is the core value. Log file analysis turns crawl budget from a theory into a measured quantity you can act on.
How to Read Googlebot Crawl Behavior
Once you have the logs and can isolate verified Googlebot, you are looking at Googlebot crawl behavior directly. A few patterns matter more than the rest.
Crawl frequency by URL. Group requests by URL and count them. Your money pages should be near the top. If high-value pages sit at the bottom and junk sits at the top, you have a crawl-allocation problem.
Crawl frequency by directory. Roll the counts up to folder level. This is where bloat becomes obvious — a /tag/ or /?sort= directory eating 40% of crawl is a flashing red light.
Status codes by volume. A healthy site sends Googlebot mostly 200s. A meaningful share of 301s, 404s, or 5xx responses means crawlers are spending your budget on redirects and dead ends instead of content.
Crawl recency. Sort by last-crawled date. Pages Googlebot has not touched in 60+ days are either low-priority in Google’s eyes or genuinely hard to reach — both worth investigating.
Finding orphan pages and crawl traps
Cross-reference the URLs in your logs against the URLs in your sitemap and your internal link graph. Two findings fall out immediately.
URLs that appear in your logs but not in your internal links are potential orphan pages — Google found them somehow, but your own site does not link to them. URLs that appear thousands of times in logs but are not in your sitemap are often crawl traps: infinite calendars, faceted filters, or session-ID URLs generating endless variations. Both are classic findings in a proper technical SEO audit, and both are nearly impossible to spot without logs.
Bot Verification: Trust Nothing by User-Agent
Here is the rule that separates real log analysis from guesswork: the user-agent string is not proof of anything. Anyone can set their user-agent to “Googlebot” in a single line of code. Fake Googlebot traffic — used for scraping, competitive intelligence, and ad fraud — is common enough that treating user-agents as truth will corrupt every conclusion you draw.
Bot verification has two reliable methods, both documented by Google.
- Reverse-then-forward DNS. Take the request IP, run a reverse DNS lookup, and confirm the hostname ends in
googlebot.comorgoogle.com. Then run a forward DNS lookup on that hostname and confirm it resolves back to the same IP. A spoofer can fake one direction but not both. Google recommends exactly this two-step method rather than trusting any single lookup. - Published IP ranges. Since 2021 Google publishes the IP blocks its crawlers use, so you can simply check whether a request IP falls inside an official Googlebot range. Bing, OpenAI, and Anthropic publish similar lists for Bingbot, GPTBot, and ClaudeBot.
A quick cross-check: if your logs show heavy “Googlebot” activity that never appears in Search Console’s verified Crawl Stats, those requests are almost certainly fake. That mismatch is one of the fastest ways to expose a spoofer.
The 2026 Shift: AI Crawlers in Your Logs
The biggest change to log file analysis in the last two years is that your access logs no longer carry a single class of bot. The same file now records three distinct populations, and confusing them leads to bad decisions.
- Indexation crawlers — Googlebot and Bingbot — crawl to build the traditional search index.
- Training crawlers — GPTBot, ClaudeBot, Google-Extended, and others — fetch content to train large language models.
- AI-search retrieval bots — the fetchers that pull live pages to answer a user’s question inside ChatGPT, Perplexity, or AI Overviews.
These have completely different implications. Blocking a training crawler is a content-licensing decision. Blocking a retrieval bot can remove you from AI answers entirely — the AI equivalent of deindexing, and directly relevant if you care about generative engine optimization. You cannot make that call sensibly until you can see, in your logs, which bots are actually requesting what.
One caution that defines the current landscape: the Perplexity stealth-crawler episode showed that some AI companies will crawl from undeclared user-agents and rotating IPs specifically to evade blocks. That makes IP-level verification more important than ever — user-agent strings are now actively adversarial, not just occasionally forged. Understanding which AI bots reach your site is also directly relevant to entity SEO and knowledge graph strategy — the crawlers that train models are building an entity graph of your brand, and what they find in your logs tells you whether they are reaching the right signals. The technical SEO service covers log analysis implementation as part of a full technical programme.
A Practical Log File Analysis Framework
Here is the step-by-step process I run on a new site. It scales from a spreadsheet on a small site to a log-analysis tool on a large one, but the logic is the same.
- Collect. Pull at least 30 days of raw, unsampled access logs across all subdomains and your CDN edge.
- Verify. Filter to confirmed bots using reverse/forward DNS or published IP ranges. Discard or quarantine everything unverified.
- Segment. Split the verified traffic by bot type — indexation, training, retrieval — so you analyse each population separately.
- Aggregate. Count requests by URL, by directory, by status code, and by day. This is where waste becomes visible.
- Cross-reference. Compare crawled URLs against your sitemap and internal link graph to surface orphan pages and crawl traps.
- Prioritise. Rank the problems by crawl volume wasted. Fixing the directory that eats 30% of crawl beats fixing ten minor issues.
- Act and re-measure. Block, redirect, or de-link the waste, then pull fresh logs in 2–4 weeks to confirm Googlebot reallocated its effort.
Run that loop quarterly on any site over ten thousand URLs and you will catch crawl problems months before they show up as ranking drops. For smaller sites, an annual pass is usually enough — and on a site of a few thousand stable pages, this is rarely your highest-leverage work. Knowing when log analysis matters is exactly the kind of judgement a focused SEO audit is meant to provide.
Tools You Can Actually Use
You do not need expensive software to start. The trade-off is volume.
For a small site, a few thousand log lines fit in a spreadsheet, and command-line tools (grep, awk, sort, uniq) handle filtering and counting fast. For larger sites, dedicated log analysers — Screaming Frog Log File Analyser, JetOctopus, Oncrawl, or an ELK/BigQuery pipeline — ingest millions of lines and join them to crawl and sitemap data automatically.
Pick the tool to match the log volume, not the marketing. The analysis logic does not change; only the scale does. If you are weighing whether to build this capability in-house or bring in outside help, that is a sensible thing to raise in any SEO consulting conversation.
What Good Looks Like
After a clean log analysis and the fixes that follow, a healthy site shows a few clear signals. Verified Googlebot spends the majority of its requests on indexable, canonical, 200-status URLs. Your most important pages are among the most frequently crawled. Status codes are overwhelmingly 200, with redirects and errors in the low single digits. And new or updated pages get crawled within days, not weeks.
If your logs do not look like that yet, you now know where to dig. Log file analysis for SEO is the closest thing the discipline has to ground truth — every other tool is interpreting the same events these files record directly. Get comfortable reading them, and you will fix crawl problems your competitors cannot even see. For context on how log analysis sits within a full technical SEO programme, the Technical SEO Hub shows how every piece connects.
Frequently Asked Questions
How often should I do log file analysis?
For large or fast-changing sites — ecommerce catalogues, marketplaces, publishers, anything over ten thousand URLs — run a full log analysis quarterly, and pull fresh logs whenever you make a major technical change like a migration or a faceted-navigation fix. Smaller, stable sites can get by with an annual review. The trigger to look immediately is any sudden change in indexing, a spike in “Discovered – currently not indexed” in Search Console, or a traffic drop you cannot explain from rankings alone.
Can I do log file analysis without technical skills?
You can get started with a spreadsheet and basic filtering, but meaningful analysis on a real site needs comfort with either command-line tools or a dedicated log analyser, plus the SEO judgement to know which patterns matter. The data extraction is learnable in an afternoon; the interpretation — knowing that a directory eating 30% of crawl is a problem and why — is where experience pays off. Many teams hand the first pass to a specialist and then maintain it internally.
What is the difference between log file analysis and a crawl with Screaming Frog?
A crawl simulates how a bot would move through your site by following links from a starting point; it tells you what is theoretically reachable and how it is structured. Log file analysis shows what bots actually requested, including URLs your internal links never point to. You need both: the crawl models intended structure, the logs reveal real behaviour. The most valuable findings come from comparing the two — URLs crawled but not in the crawl map are orphans; URLs in the map but never crawled are visibility problems.
Do AI crawlers affect my SEO?
Indirectly but increasingly. Training crawlers like GPTBot do not influence your Google rankings, but blocking or allowing them shapes whether your content trains the models people now ask instead of searching. AI-search retrieval bots are the ones that matter for visibility — if you block them, you can disappear from AI Overviews and chat-assistant answers even while ranking fine in classic search. Reading your logs is the only way to see which AI crawlers are reaching you and decide each case deliberately.
