How to Verify Googlebot Traffic With IP Range Files

A line in your server log says Googlebot just crawled your pricing page. Is it true? The user agent string says so, but user agent strings are the easiest thing on the internet to fake. Scrapers, content thieves, and competitor tools routinely dress up as Googlebot because most servers never check further than that one header. Google knows this, which is why it publishes machine-readable IP range files that let you confirm, with certainty, whether a request actually came from its crawling infrastructure.

This matters for two practical reasons. First, security: fake Googlebot traffic can scrape your content, hammer your server, or probe for vulnerabilities while hiding behind a trusted name. Second, crawl budget: if your server spends CPU cycles serving imposters, real Googlebot gets less attention, and your actual pages take longer to get crawled and indexed. Verifying bot traffic is not a paranoid edge case, it is basic hygiene for anyone running a site that depends on organic search.

Why the User Agent String Alone Proves Nothing

The User-Agent header is just a string of text the client sends with each request. Nothing stops a scraping script from setting it to Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html), the exact string real Googlebot uses. Servers that only check this header for access control or analytics segmentation are trusting a value the visitor fully controls.

The practical effect shows up in two places. In analytics, phantom “Googlebot” visits inflate crawl activity numbers and make it look like Google is spending more attention on your site than it actually is. In server logs, a burst of fake Googlebot requests can look like a legitimate crawl spike, masking what is really a scraper harvesting your content or a bad actor testing endpoints. Neither problem is visible until someone cross-checks the IP address behind the request, which is exactly what Google’s verification tools are for.

From Reverse DNS to JSON: Google’s Current Verification Methods

Google has offered a way to verify its crawlers for years, but the method has evolved.

Reverse DNS lookup, the classic method

The traditional approach is a two-step DNS check. First, you run a reverse DNS lookup on the IP address that made the request; a genuine Googlebot IP resolves to a hostname ending in googlebot.com or google.com. Second, you run a forward DNS lookup on that hostname and confirm it resolves back to the same original IP. If both checks match, the request is verified. Google still documents and supports this method, and it remains useful for one-off manual checks or for teams whose existing security tooling is built around DNS resolution rather than static lists.

The downside is speed and automation cost. Two DNS round trips per request is not something you want to do inline on every hit to a production server, especially at scale. It works well for spot-checking a handful of suspicious log entries; it is a poor fit for real-time firewall rules.

The JSON IP range files

To solve the automation problem, Google now publishes structured JSON files listing the IP ranges its crawlers use. These files can be fetched, cached, and diffed by a script, then turned directly into firewall allow-lists or server rules, no DNS lookups required at request time. This is the method worth building automation around if you manage a site with meaningful bot traffic.

googlebot.json versus special-vendors.json

Here is the detail most quick tutorials skip, and it matters: Google does not publish one file, it publishes several, and mixing them up causes real damage.

• googlebot.json lists the IP ranges used by the main crawler that indexes your pages for search, including its rendering component.
• special-vendors.json lists IP ranges for separate Google services such as Google Ads verification crawlers and other special-purpose fetchers.

If you build a firewall rule that only allows googlebot.json ranges and blocks everything else labeled “Google,” you can accidentally cut off Ads-related verification traffic, which shows up as a drop in ad quality scores or verification failures that have nothing to do with organic search. Before writing any allow-list rule, confirm which file actually matches the service you are trying to protect against impersonation.

Turning Verification Into a Security and Crawl Budget Practice

Knowing the files exist is only step one. The value comes from building a repeatable check into how you monitor your site.

A manual verification workflow

For occasional spot checks, a simple routine works: pull the suspicious IP from your server log, fetch the current googlebot.json file from Google’s developer documentation, and check whether the IP falls inside one of the listed ranges. If it does not, and the request also fails a reverse DNS check, you are looking at spoofed traffic, not Googlebot. This manual pass is worth doing whenever you see an unusual spike in “Googlebot” hits, a sudden jump in bandwidth from a single user agent, or scraped content showing up on another site shortly after a crawl.

Automating it server-side

For ongoing protection, the ranges from the JSON file can be turned into server rules. A basic Nginx approach denies all traffic claiming to be Googlebot by user agent unless the source IP matches an allowed range, something like combining a map block keyed on the real IP address with a fallback return 403 for anything outside the known ranges. Apache admins can achieve the same result with RewriteCond rules checking REMOTE_ADDR against the published ranges before allowing a request claiming the Googlebot identity through. The exact syntax depends on your stack, but the principle is the same in every case: verify the network origin before trusting the identity claim.

This is also where local crawling tools earn their keep. When you audit your own site with a crawler that runs on your machine rather than a shared cloud service, the requests in your server log are unambiguous, they come from your own IP, not a third-party pool that might get flagged or confused with bot traffic later. Seodisias works this way: it runs locally on Windows, macOS, or Linux, so when you are testing how your site responds to a crawl, you are not adding noise to the exact log data you are trying to keep clean for genuine bot verification. Reviewing how a site handles crawler access, including JavaScript rendering behavior, fits naturally into a broader technical check like the one described in our AI-ready SEO audit checklist, where bot access and rendering integrity are part of the same pass.

Why this protects crawl budget, not just security

Every request your server processes from a fake Googlebot is a request it is not spending on real crawl activity, and on larger sites, repeated scraping load can slow down response times across the board, including for the actual crawler. Blocking impersonators is not only a security measure, it is a direct way to keep your crawl budget focused on genuine Google traffic instead of being diluted by noise that looks identical at first glance but costs the same server resources.

Conclusion

Verifying Googlebot traffic is a small habit with outsized payoff. The user agent string tells you what a visitor claims to be; the IP range files, cross-checked with reverse DNS when needed, tell you what it actually is. Knowing the difference between googlebot.json and special-vendors.json keeps you from accidentally blocking legitimate Google services while filtering out impersonators. Turning that check into an automated server rule, rather than a one-time manual lookup, is what keeps the protection running after you stop paying attention to it.

If you want to see exactly how your own site behaves during a crawl, including rendering and access patterns a bot verification setup should account for, run a full audit with Seodisias, a free, local, cross-platform SEO crawler with no URL limit and no sign-up required.