Someone Is Scraping Your Prices Every Hour. Here Is How to Tell.
Price scrapers rotate through residential proxies, copy real browser strings, and request product pages in catalog order. Rate limiting alone will not stop them, and the signs are in the request pattern, not the user agent.
A competitor's pricing tool hits your product pages every hour, on the hour, in catalog order, from a thousand different residential IP addresses. Your analytics show the traffic as real shoppers, because the requests carry real Chrome user agents and residential IPs that look like households. Your server logs show the truth: nobody browses a catalog in alphabetical order at 3am, and nobody reads a product page for two seconds flat, a thousand times a day.
Price scraping is the most common bot problem on ecommerce, and the hardest to see, because the scrapers have learned to look human. The old signals, a datacenter IP, a python-requests user agent, a thousand requests a minute, catch the lazy scrapers. The serious ones rotate through residential proxies and behave just enough like shoppers to pass a casual look.
The pattern that gives them away
Scrapers are regular. They request pages in catalog order, or by product ID sequence, with consistent timing between requests, and they visit pages a human would never visit in that order, like every variant of every product, including the ones that have been out of stock for months. A human browses in bursts, with pauses, backtracking, and dead ends. A scraper browses like a list.
The second signal is what they do not do. They never add to cart, never reach checkout, never trigger the interactions a shopper triggers, and they request the price element on every page while ignoring the reviews, the images, and the related products. A session with a hundred product views and zero interactions is not a shopper. It is a harvest.
Why rate limiting alone fails
Rate limiting per IP assumes the scraper has a small number of addresses. A scraper running through residential proxies has thousands, and each one stays under the limit while the total volume stays high. DataDome, whose whole business is stopping this, puts it plainly: simple IP-based rate limiting fails against scrapers using residential proxies and rotation.
User-agent filtering fails the same way. The scraper that announces itself as python-requests is the one you catch. The one that copies a real Chrome string, which is most of them, sails through. The filter that blocks the honest bot and misses the dishonest one is worse than no filter, because it reports success while the scraping continues.
The layered defense
Stack the layers cheapest first. IP intelligence scores the source address and kills the bulk datacenter scrapers for almost nothing. Rate limiting keyed on something better than a bare IP, like a session or a device fingerprint, catches the ones that rotate. Behavioral signals, request cadence, header consistency, interaction patterns, catch the residential-proxy scrapers that survived the first two layers.
Then decide what to serve them. A hard block is one option, but a degraded response is often better: a cached price, a delayed response, or a page with the price element removed. The scraper gets stale data, the competitor's pricing tool reports a price that is wrong, and your server spends less effort than it would fighting a block the scraper routes around in an hour.
Related reading: Bot Traffic Index, Answers, Glossary.
Frequently asked questions
How do I know if bots are scraping my prices?
Look for regularity: product pages requested in catalog order, at consistent intervals, including out-of-stock variants, with hundreds of views and zero interactions. A human browses in bursts with pauses and backtracking; a scraper browses like a list.
Why does rate limiting not stop price scrapers?
Because serious scrapers rotate through thousands of residential proxy addresses, and each one stays under the per-IP limit while the total volume stays high. Rate limiting per IP assumes a small number of addresses, which is exactly what the scraper does not have.
Should I block scrapers or serve them bad data?
A degraded response is often better than a hard block. Serve a cached price, a delayed response, or a page with the price element removed, so the scraper's data goes stale. A block gets routed around in an hour; stale data poisons the competitor's pricing tool.
What are the layers of a bot defense?
IP intelligence first, to kill bulk datacenter scrapers cheaply. Rate limiting keyed on session or device fingerprint second, to catch rotation. Behavioral signals, cadence, headers, interaction patterns, third, to catch the residential-proxy scrapers that survived the first two layers.