Scrapers vs Search Crawlers: Which Bots to Block and Which to Welcome
Welcome the bots that identify themselves, read robots.txt, pace their requests, and come from published IP ranges: search crawlers and AI assistants you want citing you. Block the ones that lie about identity, rotate residential proxies, ignore robots.txt, and hammer pages. The test is behavior, and the server log is where it shows.
The split most bot advice skips
Bot traffic passed human traffic online for the first time in 2024, when automated traffic made up 51 percent of all web traffic and bad bots alone were 37 percent, per Imperva's 2025 Bad Bot Report. The following year the automated share rose past 53 percent. Those numbers get quoted as a reason to block everything, and that is the wrong conclusion. A large share of that traffic is legitimate: search crawlers indexing pages, uptime monitors checking the store is alive, price comparison services, research scrapers, and the AI assistants whose crawlers now read product pages to answer shoppers' questions. Blocking them all does not make the store safer. It makes it invisible, and the invisibility is silent, because a crawler that cannot reach a page leaves no error in any dashboard.
The real split is not good bot versus bad bot as a fixed list. It is behavior, and the behavior is visible in the server log. A legitimate crawler tells you who it is in the user agent string, arrives from IP ranges its owner publishes, reads your robots.txt file, and paces its requests so it does not knock the server over. A bad bot lies about its identity, rotates through residential proxies to look like a hundred different people, ignores robots.txt entirely, and hammers pages as fast as the server will answer. The same user agent can belong to either, because the string can be faked. The IP, the pace, and the robots.txt check are the evidence.
How to draw the line in practice
Start with the allow list, not the block list. Googlebot, Bingbot, and the AI crawlers you want citing the store, GPTBot, ClaudeBot, PerplexityBot, belong in robots.txt by name, with the paths they may crawl. Verify them by IP against the vendors' published ranges, because a scraper will happily call itself Googlebot. Then look at what is left. Requests from datacenter ASNs, AWS, Google Cloud, Hetzner, with a residential-grade browser user agent are contradicting themselves: a browser does not normally run in a datacenter, and that contradiction is the cheapest detection signal there is. Requests that hit the same product page hundreds of times an hour, or that request non-existent pages in sequence, are not shoppers. Requests that ignore robots.txt after fetching it are not legitimate crawlers.
The enforcement follows the evidence. Rate limit the datacenter traffic, challenge the suspicious patterns, and block the IPs and prefixes that keep coming back, at the /24 level where abuse clusters, not the whole ASN. And keep the good bots out of the block rules: a rule that blocks all automated traffic will eventually block the search crawler that brings the customers, and the store will find out from a traffic drop weeks later, not from an alert. The line is behavior, and the log is the only place it is written down. The measurement half is your server logs and analytics disagreeing and click fraud not being invalid traffic.
Frequently asked questions
What percentage of web traffic is bots?
Automated traffic passed human traffic in 2024 at 51 percent, with bad bots alone at 37 percent, per Imperva's 2025 Bad Bot Report. The automated share rose past 53 percent in 2025.
How do I tell a good bot from a bad one?
By behavior, not by the user agent string, which can be faked. A good bot identifies itself, comes from published IP ranges, reads robots.txt, and paces requests. A bad bot lies about identity, rotates proxies, ignores robots.txt, and hammers pages.
Should I block all bots?
No. Blocking search crawlers and AI assistants costs discovery and citations, and the loss is silent because a blocked crawler leaves no error in any dashboard. Allow the crawlers you want, by name and verified IP, and block on behavior.
What is the cheapest bot detection signal?
The ASN. A residential-grade browser user agent arriving from a datacenter ASN, like AWS or Hetzner, is contradicting itself. That mismatch is detectable before any fingerprinting, and it catches most crude bots.