Price Scrapers vs. Thieves: Sorting Legitimate Crawlers From Bad Bots
Not every scraper is an enemy. How to tell comparison-shopping crawlers apart from the bots stealing your content and margin.
Your logs show a scraper hitting your pricing pages every hour. Is it a competitor stealing your margins, or a shopping comparison site sending you customers? The answer matters enormously, and the two look similar at first glance. Here is how to tell legitimate crawlers from the bots that are actually costing you money.
Start with identity, but verify it
Legitimate crawlers identify themselves: user agent strings naming the service, reverse DNS that matches, published IP ranges. Check all three, because bad bots spoof user agents routinely. A scraper claiming to be a comparison engine from a residential proxy IP is lying.
Maintain an allowlist of verified beneficial crawlers rather than trying to blocklist the infinite supply of bad ones. Verification is a one-time cost per crawler. Blocklisting is forever.
Behavioral fingerprints
Good crawlers are polite: they respect crawl-delay, hit pages at human-plausible intervals, and request robots.txt. Price thieves are greedy: high-frequency hits on pricing and inventory endpoints, sessions that never touch a homepage or category page, requests at machine-regular intervals around the clock.
Watch the page mix. A comparison shopper's crawler reads product pages the way a shopper would, with some breadth. A scraper building a competitive pricing database hammers the same SKU set repeatedly and ignores everything else. The pattern reveals the intent.
Follow the business impact
The decisive question is what the scraper does with your data. Legitimate comparison traffic arrives as referrals you can see in analytics. Price thieves produce no referrals; the data goes into a competitor's repricer or a fake storefront undercutting you.
Quantify it. Scraping load has infrastructure costs, but the real damage is margin erosion from repricing bots and content theft enabling counterfeit listings. When the impact is real, the response budget is justified. When a crawler sends converting traffic, leave it alone.
Graduated response
Do not start with blocking. Start with rate limiting: slow abusive patterns to human speed and watch what breaks. Legitimate users never notice. Aggressive scrapers either adapt politely or reveal themselves by rotating IPs to evade the limit, which is itself a signal.
Escalate deliberately: rate limit, then serve degraded or delayed data to confirmed bad actors, then block. Each step generates evidence. Blocking first generates false positives and angry emails from partners whose crawler you misclassified.
Building the allowlist
A crawler allowlist is a living document, not a one-time setup. For each entry record the service name, the verified identification method, the IP ranges or ASNs, the pages it is allowed to hit, and the date you verified it. Review the list quarterly because services change infrastructure and old entries go stale.
Verification has three steps: check the user agent against the service's published documentation, confirm reverse DNS resolves to the claimed domain, and observe behavior for a week to confirm it matches the polite pattern. Any crawler that fails step two is not who it claims to be, regardless of what the user agent says.
Frequently asked questions
Should we block all price scrapers?
No. Some send you customers. Verify identity, check behavior, measure impact, then decide per crawler. Blanket blocking costs you legitimate traffic.
How do we handle scrapers that rotate IPs?
IP rotation to evade rate limits is strong evidence of bad intent. Fingerprint by behavior and TLS characteristics instead of IP, and escalate to blocking.
Do we need a bot management vendor for this?
At meaningful scale, usually yes. Distinguishing sophisticated scrapers from legitimate crawlers in real time is genuinely hard, and the vendors amortize that difficulty across customers.
Frequently asked questions
How is a price scraper different from a malicious bot?
A price scraper wants your public data, usually prices and stock, and often identifies itself or follows a predictable pattern. A malicious bot wants to do something: stuff credentials, test cards, or hoard inventory. The response differs, because scrapers are managed while attacks are blocked.
Can you block price scrapers without hurting real shoppers?
Rate limits, challenges on repeated rapid requests, and serving scrapers a cached public feed all reduce scraper load with little shopper impact. Blanket blocks based on rough signals tend to catch real customers, especially those on shared networks.
Why do stores tolerate some scraping?
Because prices are public anyway, and some scraper traffic comes from shopping platforms that also send buyers. The practical goal is to keep scraping cheap to serve and to stop it distorting analytics, rather than to eliminate it entirely.