A Googlebot User Agent Proves Nothing: Verifying Good Bots by Reverse DNS
Your logs show thousands of visits from Googlebot, the crawl budget looks healthy, and a share of those visits are a scraper in a rented costume. The user agent is a claim. Verification is a procedure, and Google publishes it.
The costume problem in one string
A user agent is text the visitor sends about itself, editable by anyone in seconds. Bot operators claim search engine identities for two rewards: sites allow listed crawlers through rate limits and bot defences, and log analysis treats the traffic as benign. Cloudflare's bot guidance and Google's own Search Central documentation make the same point from opposite sides of the fence: identity by user agent alone is not identity, and both companies publish verification methods because the spoofing is routine, not exotic.
The cost of believing the costume runs both ways. Allow a scraper through as Googlebot and it harvests catalogue, pricing, and content under your lightest defences. Block real Googlebot in a defensive sweep, because its behaviour looked aggressive, and the cost arrives weeks later as an indexing problem nobody connects to the firewall change. Both errors come from deciding identity on a string.
The verification procedure Google publishes
Google's method is a two step DNS check. Take the visiting IP, run a reverse DNS lookup, and confirm the hostname ends in googlebot.com or google.com. Then run a forward lookup on that hostname and confirm it returns the original IP. A spoofer can set a convincing reverse record only for IPs they control the records for, and the forward check closes the loop, which is why the pair of checks, not either alone, constitutes verification. Google also publishes its crawler IP ranges as a JSON list for operators who prefer matching at the edge.
Other legitimate crawlers publish equivalents: documented IP ranges, verification hostnames, or signed requests. The working rule for a store is a verified bots list built from those publications, refreshed on a schedule, because ranges change. A list copied from a forum post two years ago is a costume with a date on it.
Where verified identity changes decisions
Rate limiting is the first decision it cleans up. Verified search crawlers get crawl friendly limits, declared AI and SEO crawlers get the limits your policy sets for them, and unverified traffic claiming a famous name gets treated as what it measurably is: unknown traffic with a suspicious label, which is a rate limiting and monitoring candidate, not a guest of honour. Log analysis gets cleaner in the same stroke, because the search traffic line in the report stops including whoever rented a proxy that week.
Keep the humility clause in the policy. Verification proves a visit came from the operator's infrastructure. It does not prove the visit was useful, proportionate, or welcome at its observed rate, and a verified crawler crawling wastefully is still a cost to manage, through crawl controls and robots rules where the operator honours them. Identity is the start of bot management, not the end of it.
Related reading on this site: Bot Traffic Index, Answers, Glossary.
Take the ten busiest Googlebot claimed IPs in your logs and run the reverse and forward check on each. The share that fails is the size of the costume problem on your store, measured rather than guessed.
Frequently asked questions
How can I tell if a Googlebot visit is real?
Use Google's published method: reverse DNS on the visiting IP should return a hostname ending in googlebot.com or google.com, and a forward lookup of that hostname should return the same IP. Google also publishes its crawler IP ranges for matching.
Why do scrapers pretend to be Googlebot?
Because sites allow listed search crawlers through bot defences and rate limits, and treat their traffic as benign in logs. A search engine user agent is a cheap costume that buys lighter treatment on both fronts.
Does verification prove a crawler is behaving well?
No. It proves the visit came from the operator's infrastructure. A verified crawler can still crawl wastefully or against your preferences, which is managed with rate limits, robots rules, and crawl settings, not with identity checks.
How often should a verified bots list be updated?
On a regular schedule, quarterly is workable, and whenever a provider announces range changes. Crawler IP ranges change over time, and a stale list gradually misclassifies real crawlers and impostors alike.