Bots Scraping Shipping, Delivery Estimate, and Store Stock Endpoints
The quiet endpoints that answer how much, how fast, and is it in stock are a scraper's favourite data set. What they collect, why it matters, and the controls that do not break checkout.
Price pages get the security budget. The endpoints that answer shipping quotes, delivery date estimates, and per store stock levels get a shrug, and they are where a patient scraper builds a better picture of your business than your own reporting gives you. Poll the delivery estimate endpoint across postcodes and a competitor learns your courier mix, your cut off times, and the regions where your fulfilment is slow. Poll stock levels store by store, daily, and they can estimate your sell through on a product line without buying a single unit.
None of this requires breaking in. These endpoints are public because the storefront needs them, they answer politely, and at polite request rates they are nearly invisible in aggregate traffic. The defence is not to hide the shop's working parts from shoppers. It is to notice bulk, patterned interest in answers that a human asks once, and to price that interest correctly in rate limits and response design.
What the pattern looks like in the logs
A human asks for a delivery estimate once or twice, for their own postcode, at the point of buying. A scraper asks for thousands of postcodes in sequence, or every store's stock for the same product on a fixed interval, with timing that is too regular and coverage that is too complete. Look for breadth as the tell: enumeration across postcodes, store IDs, or product IDs, especially in tidy order, from infrastructure that has no matching page views. The estimate widget on the product page should generate page traffic first. Endpoint calls without the page are the smell.
Separate the actors before reacting. Search engines, price comparison services you partner with, and your own monitoring tools all touch these endpoints legitimately, and some scraping in this category is research or comparison traffic a business quietly benefits from. The line is volume, evasion, and purpose: rotating addresses to defeat limits, ignoring robots signals, and harvesting at a scale whose only use is a competing data set. Draw the line there, in writing, or every block decision becomes an argument.
Controls that raise the cost without taxing shoppers
Rate limit by behaviour, not just by address. A budget of estimate calls per session, generous for a human comparing two addresses and absurd for the ten thousandth lookup, catches enumeration without touching checkout. Cache aggressively at the edges you control: delivery estimates for a postcode and basket type change rarely, and serving a cached answer costs you nothing while giving a scraper yesterday's news at scale, which degrades the data set they are building in a way they may not notice quickly.
Coarsen the answers where precision is not needed. A delivery window of 2 to 4 working days tells a shopper what they need and tells a harvester far less than an exact date per postcode. Stock shown as in stock, low, or out, rather than exact unit counts per store, is the same principle, and most retailers already display it that way on the page while leaving exact counts in the endpoint response for anyone who asks. Match the endpoint's honesty to the page's. The page is the policy.
Measure it as a data leak, not a traffic stat
Report endpoint scraping alongside traffic quality metrics, not as raw blocked counts for applause. The useful numbers are coverage attempts detected, the share of estimate calls with no matching session, and whether the pattern returns after controls. A rival who stops enumerating your stock endpoint after you coarsen it has told you the data was worth having, which should inform what else you review: returns windows, price match rules, and any other public answer a competitor could poll into a strategy.
Keep the incident notes factual and sourced, in the style this beat respects: dates, volumes, the endpoint, the behaviour observed, the control applied, and the result. Vague claims about bot problems convince nobody in a budget meeting. A short log showing a daily full catalogue stock harvest, stopped by two specific changes, funds the next piece of work by itself.
Related reading: Bot Traffic Index, Answers, Glossary.
Frequently asked questions
Why would anyone scrape delivery estimates?
Delivery speed and cost win sales, so a competitor who knows your courier performance by region knows where you are beatable. The estimate endpoint reveals cut offs, regional speed, and shipping pricing at a scale no mystery shopper could match, simply by asking politely for thousands of postcodes.
Is all scraping of these endpoints malicious?
No. Search crawlers, comparison partners, researchers, and your own monitoring all use them, and some comparison exposure benefits a store. The problem pattern is high volume enumeration, evasion of limits, and harvesting whose scale fits building a rival data set rather than answering a shopper's question.
What is the first control to add?
A session based rate budget for estimate and stock calls, generous enough that no real shopper meets it, plus edge caching of repeated answers. Then coarsen responses to match what the page displays: windows instead of exact dates, stock bands instead of unit counts.
How can we tell if a scraper is a competitor?
Often you cannot attribute it with certainty, and the report should say so. What you can document is the behaviour: coverage, timing, volume, and evasion. Controls and coarsening do not need an attribution to be justified; the data exposure stands on its own.