Bots at the API Layer: Probing Product, Search, and Stock Endpoints
Modern storefronts expose JSON endpoints for search, stock, and pricing that bots query directly, skipping pages entirely. What API probing looks like in logs and what it distorts.
The page view numbers look calm, which is why the team stopped worrying. Underneath them, the store's own search and stock endpoints were being queried thousands of times an hour by clients that never render a page: the storefront's JSON layer, built to make filtering feel instant, is also a clean machine readable catalogue of prices, inventory, and variants, and scrapers have noticed.
Bot measurement built on pages misses the API layer entirely. This post explains which endpoints get probed, how the traffic presents in logs (regular intervals, complete variant coverage, no image or script requests), and what it costs: not just bandwidth, but inventory intelligence handed to competitors and resellers on a schedule.
What the probing pattern looks like
Page scraping is messy; endpoint probing is tidy. Requests arrive at exact intervals, walk product identifiers or search terms in sequence, and request every variant combination rather than the two a human compares. Sessions do not exist here: there are no page loads, no assets, no cookies being returned, often just a rotating set of IP addresses with a plausible browser user agent that never asks for a single image, which no real browser has ever done.
The index value of watching this layer is that intent is legible. A client enumerating stock levels across the catalogue hourly is building an inventory feed, usually for resale arbitrage or competitor monitoring. A client hammering a discount or gift card endpoint is doing something worse. The endpoint being hit tells you the business model on the other end, which should decide how much you care.
Defence without breaking the storefront
The endpoints exist to serve your own front end, so they cannot simply be closed. The working controls are rate shaping per client, requiring the session context the real storefront establishes, and degrading gracefully: a client that exceeds a human plausible query rate gets cached, delayed, or partial data rather than errors that teach it the threshold. Stock precision is a dial you own, too: showing low stock as a threshold ('only a few left') to unauthenticated API callers, while keeping exact counts for the cart's own needs, removes much of the resale value of the feed.
Measure the layer separately in your bot reporting. A single index number for the whole property hides the split between page bots, which mostly distort analytics, and API bots, which mostly leak commercial information. They are different problems with different owners, and averaging them serves neither.
Related reading: Bot Traffic Index, Answers, Glossary.
Frequently asked questions
Why do bots prefer API endpoints over scraping pages?
Endpoints return clean structured data, prices, variants, stock counts, without the cost of rendering pages or parsing HTML. A storefront's search and product JSON is faster to query, cheaper to run at scale, and more reliable to parse than the visible site, which is exactly why it attracts systematic probing.
How can we spot API probing in our logs?
Look for clients requesting endpoints at regular intervals, enumerating product or variant IDs in sequence, with no accompanying page, image, or script requests, and no persistent cookies. Human shoppers generate messy, asset heavy sessions. A tidy client that only ever asks for data is a machine by definition.
Does API bot traffic affect our analytics?
Mostly not directly, since it never loads pages or analytics tags. Its costs are elsewhere: infrastructure load, and competitors or resellers gaining a live feed of your pricing and inventory. That is why it belongs in bot reporting as its own category rather than being dismissed because dashboards look clean.
Can we just block data centre IPs from the endpoints?
It helps with the crudest traffic, the general invalid traffic tier, but systematic operators route through residential proxies that look like customers. Rate shaping, session requirements, and reducing the precision of public stock data catch what IP lists miss, without blocking real shoppers sharing those networks.