Telling Search Crawlers, Shopping Agents, and Harmful Bots Apart in Your Logs
Not all automated traffic is an attack, and not all of it is welcome either. A log reading guide for separating crawlers, agents, and abuse.
Server logs lump every automated visitor into one suspicious pile, and teams respond with one blunt tool: blocking. That approach breaks search visibility when a crawler is blocked, loses customers when a shopping agent is blocked, and still misses sophisticated abuse that rotates its identity faster than rules update.
Classification beats blocking. This guide covers the practical signals that separate beneficial crawlers, legitimate shopping agents, and harmful automation, and the handling each class deserves.
Verify declared crawlers before trusting or blocking them
Major search crawlers declare themselves in their user agent and can be verified: check that the visiting address belongs to the operators published ranges or resolves to their domain, and that it behaves like a crawler, steady pace, robots file respected, pages in a crawl like order.
Anything using a crawlers name without passing verification is not a crawler. It is an impersonator, and impersonation is itself a strong abuse signal, because legitimate automation has no reason to lie about what it is.
Recognise shopping agents by task shaped behaviour
Shopping agents act on a persons behalf: they fetch product, price, and availability information, often identify themselves as agents, respect rate limits, and sometimes proceed into a normal purchase path. Their sessions look like a focused shopping task, because that is literally what they are.
Give identified agents a clean path: accurate product feeds and structured data reduce their need to scrape, and a monitored allowance beats a block that pushes the same demand into disguised, heavier scraping. Log their visits and purchase outcomes as a channel, however small it starts.
Spot harmful automation by behaviour, not identity
Harmful bots change user agents and addresses routinely, so identity lists decay within weeks. Behaviour endures: credential testing patterns, rapid sequential requests across unrelated products, gift card and discount endpoint probing, checkout attempts with systematically varied details, and complete indifference to robots rules and rate limits.
Score behaviour across a session rather than reacting to single requests. One fast request proves nothing. A session pattern of probing, testing, and evasion is strong evidence, and scoring it keeps defences calm during genuine traffic spikes that superficially resemble attacks.
Set a handling policy per class and review it monthly
Write the policy down: verified crawlers allowed and monitored, identified shopping agents allowed with rate limits and logging, unverified impersonators challenged, behaviour scored abuse blocked. When everyone on call knows the classes, incident responses stop being improvised at 2am.
Review classifications monthly against outcomes: crawl coverage in search tooling, agent referred orders, and abuse incident counts. Misclassification is inevitable at the edges. A monthly review is what stops a small edge error becoming a permanent, invisible tax on visibility or sales.
Related reading
If this topic is useful, these earlier guides go deeper on connected questions: Bots Targeting Gift Card Balance Checks, Bot Traffic Around Loyalty Points Sales and Bonus Events, and Price Scraping Bots: Reading the Traffic and Protecting Your Catalogue.
Frequently asked questions
Why not just block all bots?
Because the category includes search crawlers that bring your customers and shopping agents acting for real buyers. Blanket blocking trades visibility and sales for a quiet log, while determined abuse rotates identity and returns anyway.
How can a search crawler be verified?
By checking the visiting address against the operators published ranges or reverse domain, and by behaviour: steady pace, robots rules respected, and crawl like page patterns. Name alone is not verification, because names are copied.
What behaviour most reliably signals harmful bots?
Probing and testing patterns across a session: credential attempts, endpoint enumeration, rapid unrelated requests, and systematic variation at checkout, combined with ignoring rate limits. Single requests prove little. Session patterns prove a lot.
How often should traffic classification be reviewed?
Monthly, against search visibility, agent referred orders, and abuse incidents. Automated traffic evolves quickly, and a policy written once and never reviewed drifts out of alignment with what is actually visiting.