Distinguishing Search Crawlers From Content Scrapers by Behaviour
Both fetch a lot of pages. One follows your rules, identifies itself, and sends you customers. The other rotates identities and takes the catalogue. Telling them apart on behaviour, not user agent alone.
Every large site is crawled constantly, and the traffic is not one thing. Search engine crawlers fetch pages in order to index them, and the value exchange is obvious. Content scrapers fetch the same pages to republish a catalogue, undercut prices in real time, or stock a dataset, and the value flows in one direction only. Treating the two as a single category called bots leads to the two expensive errors: blocking the crawler that sends you customers, or welcoming the scraper that is copying the store.
User agent strings alone will not separate them, because claiming to be a search crawler costs a scraper nothing. Behaviour separates them, and behaviour is harder to fake at scale, which is why serious bot work is built on it.
What a legitimate crawler actually does
A genuine search crawler identifies itself consistently, publishes the IP ranges it crawls from so its traffic can be verified by reverse DNS, respects robots.txt directives and crawl delays, and fetches at a rate that is heavy but patterned. Verify the claim properly: take the IP, check the reverse lookup resolves to the search engine's domain, then confirm the forward lookup returns the same IP. A scraper borrowing the name usually fails that test in seconds.
Its fetching also has a shape. It revisits pages according to how often they change and how important they are, it follows links, and it does not systematically harvest every product variant, every paginated archive, and every faceted navigation combination in catalogue order on a first visit. That exhaustive, sequential completeness is a scraper's signature, because a scraper is building a copy and cannot afford gaps.
What a scraper does when it is trying not to look like one
Residential proxy rotation is the standard disguise: requests spread across thousands of household IP addresses, each making few requests, so per IP rate limits see nothing. The giveaway moves up a level, to the aggregate pattern. The full catalogue still gets fetched, in order, at a total rate no human population produces, often with no image, CSS, or script requests alongside the HTML, because the scraper wants text and prices, not a rendered page.
Session behaviour tells the same story. Real visitors and real crawlers have different but internally consistent patterns. Scraping traffic shows no add to cart, no search refinement, no account activity, just page after page of product data, frequently hitting structured endpoints, feeds, and sitemap files first to build its target list. A client that fetches your entire sitemap in thirty seconds and then works through it methodically has introduced itself accurately, whatever its user agent says.
Draw the line in a policy, not in a panic
Decide in advance what is allowed, because not all scraping is an attack. Price comparison, research, accessibility tooling, and a search engine you have not heard of yet all fetch pages automatically, and a blanket hostility to automation will catch them alongside the catalogue copier building a competing store. The defensible line is use: indexing and referral are an exchange, wholesale republication and real time undercutting are extraction.
Then enforce the line proportionally. Challenge or slow extraction pattern traffic, block verified impersonation of search crawlers outright, and keep allowing verified crawlers the access your robots.txt promises them. Review the logs monthly for new high volume fetchers and classify them deliberately. The sites that get scraped worst are usually the ones that never looked, because a scraper facing no response has no reason to stop or even to slow down.
Related reading: Glossary, About, Credential Stuffing Patterns on Store Accounts and What They Look Like.
Frequently asked questions
Why not trust a crawler's user agent string?
Because it is free to fake. Scrapers routinely claim to be major search crawlers. Verification requires checking that the IP belongs to the claimed operator, using reverse and forward DNS lookups against the ranges the operator publishes.
What behaviour most reliably identifies a catalogue scraper?
Exhaustive, sequential fetching of the whole catalogue and its variants, often after harvesting the sitemap, with no rendered assets and no shopping behaviour. A copy cannot afford gaps, and that completeness is hard to disguise.
How do scrapers evade per IP rate limits?
By rotating through residential proxy networks so each address makes few requests. The pattern is still visible in aggregate, in total fetch rate and catalogue coverage, which is why detection has to look above the single IP level.
Is all scraping harmful?
No. Search indexing, price comparison, and research involve automated fetching with a value exchange. The line is use: wholesale copying of a catalogue for republication or real time undercutting is extraction and can be challenged and blocked on that basis.