Reading Bot Traffic in Search and Filter Navigation Logs
Faceted navigation generates endless URL combinations, and scrapers know it. Here is how to read search and filter logs to tell indexing, price monitoring, and inventory scraping apart.
Facets are a bot buffet. Colour times size times price band times sort order produces more URLs than any store has products, and automated traffic works through them systematically, because each combination answers a slightly different question about price and stock. Human shoppers use two or three filters and stop. The logs show the difference clearly, if anyone reads them by session instead of by pageview.
Start by separating the three intents you will actually find: legitimate crawling that respects a pattern and a rate, commercial monitoring that wants prices on a schedule, and extraction that wants the catalogue and its stock levels as a dataset. They overlap in method and differ in tolerance. Your response should differ too.
Related reading on this site: Bot Traffic Index, Credential Stuffing Patterns on Store Accounts and What They Look Like, Bots Inflating Add to Cart Metrics and How to Spot Them.
Session shape tells you more than user agent
Look at combinations per session, depth of pagination, and whether the session ever touches a product page with dwell, a cart, or an asset a script would skip. A session that requests four hundred filter combinations, sorted every way, with no images loaded and metronome timing, is enumerating. A session that checks the same fifty product URLs every morning is monitoring, and it is usually the easier conversation.
User agent strings help at the margins and lie at the centre. Declared crawlers can be verified against their published ranges and reverse DNS. Undeclared agents claiming to be a current browser, hitting facets from a data centre network at machine intervals, should be judged on behaviour. That is the honest taxonomy, and it is the one that holds up when challenged.
Protect the catalogue without breaking discovery
Facets need to stay crawlable where they create real landing value, and they need restraint everywhere else. Cap parameter combinations in robots guidance where the platform allows, noindex the thin combinations, cache aggressively so enumeration costs you less, and rate limit by behaviour at the edge rather than by blocking whole networks you have not verified.
Be careful with the blunt fix. Blocking data centre traffic wholesale will catch researchers, accessibility tools, feed fetchers your partners rely on, and the search crawlers checking from unexpected ranges. GIVT and SIVT thinking applies here too: recoverable waste from declared crawlers is a different problem from covert extraction, and collapsing them produces confident, wrong decisions.
Keep a baseline, or every spike looks like an attack
Record normal automated share on facet URLs by week. Promotions, launches, and press coverage all move it, because legitimate monitoring intensifies when prices move. With a baseline, a spike during your sale is expected monitoring. The same spike on a quiet Tuesday, aimed at stock levels on scarce products, reads very differently.
Name what you find in the log review, specifically: the network, the pattern, the pages, the rate. An index that reports bots in general terms will be ignored. One that shows a named source enumerating a named category, with dates, gets a control changed.
Frequently asked questions
How can I tell a scraper from a keen human shopper in filter logs?
Look at session scale and rhythm. Humans apply a few filters, view products, and behave irregularly. Enumeration shows hundreds of combinations, systematic sorting and pagination, steady timing, and often no image or asset requests. Judge the session pattern, not a single request.
Should all bot traffic on facets be blocked?
No. Declared search crawling and some monitoring are part of operating a public store. Reserve blocking for covert extraction, abusive rates, and behaviour that degrades the site, and verify declared crawlers instead of trusting their user agent string.
What is the cost of leaving facet scraping unchecked?
Server load is the visible cost. The quieter costs are catalogue and pricing intelligence handed to competitors on schedule, and analytics pollution, where automated filter sessions distort conversion and search demand data that buying decisions rely on.
Do rate limits stop determined scrapers?
Rate limits raise cost and slow extraction, especially combined with caching and behaviour based challenges. Rotating residential proxies blunt simple limits, which is why controls are layered and aimed at behaviour, not a single signal.