Search Console's Crawl Stats report tells you Googlebot made a certain number of requests last week, broken down by response code and file type, in aggregate, with a delay of a day or two. What it doesn't tell you is which specific URLs got hit, how often, or whether Googlebot spent half its budget on faceted navigation parameters nobody cares about while your new product pages sat unvisited. For that, you need the one dataset Google doesn't summarize for you: your own server logs.
Log file analysis has a reputation for being an advanced, infrastructure-heavy discipline reserved for sites with tens of millions of URLs. In practice, the workflow is straightforward once you've built it once, and the insight it produces (what Googlebot is actually doing on your site, not what a sampled report estimates it's doing) is hard to get any other way.
Photo by Polina Zimmerman on Pexels
Why Crawl Stats in Search Console Don't Tell the Full Story
Search Console's crawl data is aggregated, sampled, and delayed. It groups requests by response code and resource type, which is useful for spotting a spike in 404s or a sudden drop in total crawl requests, but it won't tell you that Googlebot crawled the same six sort-order variants of your category page 40 times this week while your cornerstone content pages went untouched.
The report also can't distinguish between Googlebot proper and the dozen other crawlers, some legitimate and some not, that identify themselves with similar user agent strings. A raw access log, pulled directly from your web server or CDN, has none of these limitations. Every request is there, with the exact URL, timestamp, response code, and user agent string, unfiltered and unsampled.
What a Raw Server Log Actually Contains
A standard web server log line records the requesting IP address, a timestamp, the HTTP method and requested path, the response status code, the size of the response, and the user agent string, typically in Common Log Format or the slightly richer Combined Log Format that also captures the referrer.
That's already enough to answer the core question this workflow exists for: which URLs is Googlebot actually requesting, how often, and what response code did each request get. Some setups add extra fields, response time being the most useful addition, since a URL that consistently takes three seconds to respond is a URL Googlebot will crawl less of over time as it adapts to your server's apparent capacity.
Getting Access to Log Files in the First Place
The first obstacle is often organizational, not technical. On a traditional web server (Nginx, Apache), logs are usually already being written somewhere, and the question is just getting read access or an export from whoever manages the infrastructure. On a CDN-fronted site, most requests never reach the origin server at all, so you need the CDN's own edge logs instead, which Cloudflare and most other CDN providers expose through a logging or analytics API rather than a flat file on disk.
Retention matters here too. Many teams discover their log retention window is seven or fourteen days, which is enough to spot an acute problem but not enough to compare crawl patterns before and after a site change made a month ago. If log file analysis is going to be a recurring practice rather than a one-time emergency audit, it's worth having the retention conversation with whoever owns the infrastructure before you need a longer window and don't have it.
Filtering Out Everything That Isn't Googlebot
Raw logs contain requests from real users, from dozens of other crawlers (Bingbot, various SEO tool crawlers, scrapers, and plenty of outright bots pretending to be something else), and from Googlebot itself. The first processing step is always filtering down to just the user agent strings containing "Googlebot," which typically drops the log volume by well over 90 percent before any real analysis starts.
It's worth keeping the different Googlebot variants distinct rather than collapsing them into one bucket. Googlebot Smartphone, Googlebot Image, Googlebot Video, and the standard desktop Googlebot each crawl somewhat different subsets of a site, and a pattern that looks fine in aggregate can hide the fact that the smartphone crawler, the one that actually matters most for mobile-first indexing, is barely reaching your product pages at all.
Photo by Towfiqu barbhuiya on Pexels
Verifying You're Looking at Real Googlebot, Not a Spoofed User Agent
A user agent string is just a text field the client sends, and it's trivial to fake. Plenty of scrapers and low-quality bots set their user agent to "Googlebot" specifically because it gets past naive filtering and sometimes gets preferential treatment from rate limiters. Filtering on user agent string alone will pull in a meaningful amount of noise from these impostors.
The reliable check is a reverse DNS lookup on the requesting IP address, confirming it resolves to a googlebot.com or google.com subdomain, followed by a forward lookup on that resolved hostname to confirm it maps back to the original IP. Google's own documentation covers this verification method in detail, and it's worth running as an automated step in whatever script or pipeline processes the log file, not a manual spot check done once and assumed to still hold.
Finding Crawl Budget Waste: Parameters, Duplicates, and Dead Ends
Once you have a clean, verified set of real Googlebot requests, the pattern that shows up most often on mid-size and large sites is parameter waste: the same underlying page, crawled repeatedly under different sort, filter, or session-tracking query strings, each treated by Googlebot as a distinct URL until your canonical tags or robots rules tell it otherwise.
Grouping requests by path while ignoring the query string, then comparing that against the raw per-URL request count, surfaces this fast. A product listing page that gets 40 raw requests but collapses to 3 distinct paths once you strip parameters is a page family that's eating far more crawl budget than its actual content footprint justifies. The same grouping exercise tends to surface redirect chains too: URLs that 301 to another URL that 301s again before landing on the final destination, each hop costing a crawl request that could have gone toward a page that actually needs indexing.
Comparing What Googlebot Crawls Against What You Want Indexed
The most useful single output of this whole workflow is a simple comparison: take your XML sitemap or a full URL export of the pages you actually want indexed, and diff it against the distinct paths Googlebot requested in your log window. Two gaps matter here, and they point to different problems.
Pages in your sitemap that never show up in the crawl log at all suggest a discovery problem, weak internal linking, orphaned pages, or a robots directive quietly blocking access. Pages that get crawled heavily but aren't in your sitemap or intended indexing set are usually the parameter and redirect waste described above, consuming budget without contributing anything you actually want ranked.
"The sitemap tells Google what you want it to look at. The log file tells you what it's actually looking at. When those two lists diverge significantly, that gap is usually where the real crawl budget problem lives, not in whatever Search Console happens to be sampling that week." - Dennis Traina, founder of 137Foundry
Tools That Make This Workflow Repeatable
Building this from raw log parsing scripts is a reasonable starting point, but a handful of tools make the recurring version of this workflow far less painful. Screaming Frog's Log File Analyser is purpose-built for exactly this use case, importing raw logs and automatically separating verified Googlebot traffic, cross-referencing it against a crawl or sitemap, and surfacing orphaned and wasted URLs without custom scripting.
For teams that want to build this into existing infrastructure rather than adopt a dedicated SEO tool, GoAccess is a fast, open source log analyzer that runs directly against Nginx or Apache log formats, and the Elastic Stack is a common choice when log volume is high enough that a team wants Googlebot crawl data queryable alongside the rest of their observability data rather than living in a separate one-off tool.
Turning Findings Into Fixes
The analysis itself doesn't fix anything, and the most common failure mode of a log file audit is a well-organized spreadsheet of findings that nobody acts on. Parameter waste usually gets addressed through a combination of canonical tags pointing at the clean URL, robots.txt rules excluding known-noisy parameter patterns, and, where feasible, actually removing the parameter-generating links from internal navigation rather than just telling Googlebot to ignore what it finds.
Discovery gaps, pages that should be crawled but aren't, get fixed through internal linking rather than sitemap submission alone. A sitemap entry is a suggestion; a real internal link from a page Googlebot already crawls regularly is a much stronger signal that a URL deserves attention. If your internal linking structure is thin, 137Foundry's technical SEO service works through exactly this kind of crawl-to-architecture gap analysis for sites where the sitemap and the actual crawl behavior have drifted apart.
Building This Into a Recurring Process, Not a One-Time Audit
A single log analysis run is useful for catching an acute problem, but the real value shows up when it's repeated on a schedule, monthly for most sites, weekly for large ones with frequent structural changes, so that a new parameter pattern or a broken redirect chain gets caught within weeks instead of surfacing six months later as a slow ranking decline nobody can immediately explain.
Treat the sitemap-versus-crawl-log comparison as the recurring health check, and the deeper per-URL analysis as something you run when that comparison flags a gap worth investigating. Most of the workflow described here can be scripted once and rerun automatically, which is the difference between log file analysis being a specialized audit performed occasionally and it being a normal part of how a team monitors its own site.
If you're evaluating whether your site's crawl behavior matches what you actually want indexed, 137Foundry works through this kind of technical SEO audit regularly. You can see the rest of what we offer, read more about how we work, or start at the 137foundry.com homepage for more writing like this.
