Bots Scraping Inventory and Price Feeds: What the Traffic Means
Price comparison and competitor scrapers harvest store feeds and product pages at scale. How to read scraper traffic and decide what to block, allow, or price for.
Some bot traffic wants your data, not your checkout. Scrapers harvest prices, stock levels, and assortment from product pages and feeds, often politely, sometimes at volumes that distort analytics and slow real shoppers during peaks.
Not all scraping is an attack, and not all of it should be blocked. This guide covers recognising feed and price scrapers in traffic, measuring their cost, and choosing responses by type rather than treating every bot as one problem.
Recognise the scraper pattern
Price scrapers move like librarians: product pages in catalogue order, category pages as indexes, regular intervals, no basket, no checkout, and a hit rate on obscure products no human audience produces. Feed scrapers go straight for feed URLs and structured endpoints repeatedly.
Check the client details and the network behind the traffic. Established comparison services often identify themselves and arrive from stable ranges. Unattributed scrapers rotate addresses and browser identities, which is itself the most useful signal about intent.
Measure cost before reaching for blocks
Scraper traffic costs three things: infrastructure load, analytics distortion, and competitive exposure. The first two are measurable: share of requests, share during peak, and how much of your reported product view traffic is not human.
Competitive exposure needs a judgement call. Public prices are public. The question is whether realtime, complete, machine readable harvesting at scale changes your position, for example where dynamic pricing by competitors reacts to your prices within the hour.
Respond by category, not by reflex
Known comparison and shopping services that send you customers are usually allowed, monitored, and held to rate limits. Your own affiliate and partner feeds are contracts, not traffic problems, and should run through authenticated feeds rather than page scraping.
Anonymous high volume scrapers get rate limits first and blocks when limits are ignored or identities rotate to evade them. Rate limiting preserves legitimate uses, search engines, accessibility tools, uptime monitors, that a blunt block would cut off.
Offer a feed and watch who still scrapes
A documented product feed for partners converts scraping into a relationship: fresher data for them, attribution and control for you. Partners who move to the feed are identified. Traffic that keeps hammering pages anonymously after a feed exists has told you what it is.
Review scraper traffic monthly alongside the rest of the bot picture. Volumes shift with your prices, your competitors tooling, and the shopping calendar, and a policy set during a quiet month meets November unprepared.
Related reading
If this topic is useful, these earlier guides go deeper on connected questions: Bots and Gift Card Balance Checking Attacks, Bot Traffic in Waitlist and Back in Stock Signups, and Bot Traffic During Early Access and Presale Events.
Frequently asked questions
How can I tell scrapers from ordinary bots?
Look for catalogue order browsing, full range coverage including obscure products, strict regularity, and no basket or checkout behaviour. Identified services use stable networks. Evasive rotation of addresses and identities signals uninvited harvesting.
Is price scraping illegal or always harmful?
Prices on public pages are public, and some scrapers, comparison services, send real customers. The harm calculus is load, analytics distortion, and realtime competitive exposure, which is why responses should differ by scraper type.
Should I block all scraping?
No. Rate limit anonymous high volume traffic, allow identified partners and services that return value, and supply an official feed. Blunt blocks also catch search engines, monitors, and accessibility tools.
Why offer an official product feed?
It gives partners better data with attribution and control, and it separates friends from strangers: whoever keeps scraping pages anonymously after a feed exists has identified their intent.