Price Scraping Bots: Reading the Traffic and Protecting Your Catalogue
Competitors and aggregators harvest your prices daily. How to tell scraping from shopping, and what defence does not hurt real buyers.
Somewhere in your traffic is a visitor who views four hundred product pages an hour, never touches the cart, and knows your prices better than your merchandising team does. Price scraping is the most tolerated bot activity in commerce, because blocking it feels risky and the harm feels abstract. Neither feeling survives a look at the data.
This guide covers recognising scraping patterns in your own traffic, the real costs beyond server load, and defences calibrated to raise scraper costs without taxing genuine shoppers.
Recognising a scraping pattern
Scrapers are systematic: catalogue order browsing, complete category coverage, regular intervals, and session patterns no human shopper produces. They read listing and product pages and skip images, reviews, and everything a person lingers on. In aggregate, a scraping operation looks like a very fast, very thorough reader who never buys.
Separate price scraping from search engines and legitimate aggregators you actually feed. The test is behaviour plus identity: verified crawler traffic following your official feeds wants your data with permission. Unidentified systematic harvesting, rotating addresses to dodge limits, wants it without. The response should differ accordingly.
The real costs of being scraped
Server load is the smallest cost. Scraped prices arm competitor repricing tools, so your promotions get matched within hours and your price testing leaks its results before you have read them yourself. Aggregator sites built on scraped catalogues can outrank your own pages for your product names, selling your assortment back to your customers with someone else's margin attached.
There is also an accuracy cost. Scrapers cache, so the market sees your old prices, including errors, long after you fix them. Stores discover this when a customer demands a price the store stopped offering a week ago, quoted from a comparison page the store never supplied.
Defence that does not punish shoppers
Rate shape the catalogue, not the customer: sensible limits on sustained page rates, challenges triggered by systematic patterns rather than by geography or device, and your best prices kept accurate in the official feeds you control so legitimate channels have no reason to scrape you.
Vary what is expensive to harvest. Structured promotions, member pricing shown after login, and bundle economics are all harder to reduce to a single comparable number than a flat public discount. None of this requires hiding prices from shoppers. It requires making bulk extraction cost more than it returns.
Decide your policy before you need it
Write down which harvesting you permit, which partners get feeds, and what triggers a block, before an incident forces the decision at speed. A scraping policy owned by one person, reviewed when a new aggregator appears in the data, prevents both extremes: ignoring industrial scale extraction and blocking a partner channel in a panic.
Review the traffic monthly with fresh eyes. Scrapers adapt to defences, rotate infrastructure, and slow down to blend in. The pattern that identifies this month's operation will not be last month's signature, which is why the review is a habit and not a project.
Related reading
If this topic is useful, these earlier guides go deeper on connected questions: Credential Stuffing Bots Are Flooding Your Login Pages, Bots Are in Your Internal Site Search Logs Too, and Bot Traffic From Price Comparison and Reseller Scrapers.
Frequently asked questions
How can I tell scraping from normal browsing?
By system and scale: complete catalogue coverage in order, hundreds of product views per session with no cart activity, regular timing, and no engagement with images or reviews. Legitimate crawlers identify themselves and follow your feeds. Scrapers rotate identities to avoid limits.
What does price scraping actually cost a store?
Competitor repricing that neutralises promotions within hours, leaked price test results, aggregator pages outranking yours on your own products, and stale cached prices being quoted back to you by customers. Server load is usually the smallest line in that list.
How do I slow scrapers without hurting shoppers?
Rate shape sustained catalogue level browsing, trigger challenges on systematic patterns rather than on who the visitor is, and keep official feeds accurate for legitimate partners. Member and bundle pricing structures also raise extraction cost without hiding prices from real buyers.
Should I block all scraping?
No. Verified search and partner traffic you have chosen to feed is distribution, not theft. Block the unidentified, limit dodging, industrial harvesting. A written policy distinguishing the two, owned and reviewed monthly, prevents both neglect and panic blocking.