Not All Bots Are Waste: Separating Search Crawlers and Monitoring From Scraping

Blocking everything automated breaks search indexing, uptime checks, and partner feeds. A taxonomy that protects the store without cutting off the crawlers it needs.

The laziest bot policy is block everything that is not a human, and the first casualty is usually the store itself: search engine crawlers that keep products indexed, uptime monitoring that warns you the site is down, feed fetchers that keep your listings alive on other channels. Teams discover this weeks later, in falling organic traffic, and call it an algorithm update.

Practitioners split automated traffic by identity and behaviour, not by the single fact of being automated. GIVT and SIVT in ad measurement, general and sophisticated invalid traffic, exist because the category error costs money in both directions.

Verify identity, do not trust the label

Any bot can claim to be a search crawler in its user agent string, and scrapers routinely do. Verification is a technical check: reverse DNS on the IP, confirming it resolves to the operators domain, then forward confirming it back to the same IP. Crawlers from the major search engines pass this. A scraper wearing the label does not.

Keep an allowlist built on verified identity and published IP ranges where operators provide them, and review it. Partners, monitoring services, and feed platforms change infrastructure, and an allowlist written once and never maintained quietly becomes a list of outages waiting for their moment.

Judge the rest by behaviour

For everything unverified, behaviour is the evidence: request rate, catalogue coverage pattern, whether it respects robots directives, whether it executes the page like a browser or fetches raw documents in sequence, and what it does with what it takes. Price scraping at catalogue scale, account creation attempts, and checkout probing each have signatures that legitimate monitoring never produces.

Be honest about the grey zone. Price comparison and research crawling exist, some of it benefits the store, and a policy that treats all scraping as theft will not survive contact with the partnerships team. Draw the line at scale, deception, and load, and write it down so enforcement is consistent rather than mood based.

Measure before and after any block

Before tightening rules, baseline what verified good bots fetch and how often, and what organic landing traffic depends on it. After a change, watch crawl stats and index coverage the way you would watch checkout after a payment change. The failure is silent, delayed, and routinely misdiagnosed as a search update.

Bot management is a taxonomy exercise with a budget attached. Get the categories right, verified good, unverified and behaved, unverified and abusive, and the tooling decisions mostly make themselves.

Related reading

If this topic is useful, these earlier guides go deeper on connected questions: Scrapers That Hit Size Guides, Stock Feeds, and the Quiet Corners, Reading Bot Traffic in Search and Filter Navigation Logs, and Credential Stuffing Patterns on Store Accounts and What They Look Like.

Frequently asked questions

How do I verify a search crawler is genuine?

Check the IPs reverse DNS resolves to the search operators domain, then forward confirm that domain back to the IP. A user agent string alone proves nothing, because scrapers copy it.

What is the difference between GIVT and SIVT?

General invalid traffic is readily identified automation such as known crawlers and datacentre traffic. Sophisticated invalid traffic is automation designed to mimic humans and evade detection. The split matters because they need different responses.

Should all scraping be blocked?

Not automatically. Some crawling, such as price comparison and research, can benefit a store. Policy should turn on scale, deception, and load, and should be written down rather than improvised.

What should be monitored after tightening bot rules?

Crawl activity, index coverage, and organic landing traffic, alongside the abusive traffic you meant to stop. Damage to good bots is silent and often misread as a search algorithm change.