How Bot Traffic Distorts A/B Tests on Stores
Bots do not split evenly across test variants, and they never buy. How automated traffic quietly bends A/B test results and what to filter.
A/B tests assume the visitors in each variant are comparable humans. Bot traffic breaks that assumption twice: bots distribute unevenly, because some variants load faster or block less, and bots behave in ways no human does, instant bounces, perfect scrolling, and zero purchases, diluting conversion in whichever variant they favour.
The distortion is rarely dramatic enough to look wrong. It is a few percentage points of conversion difference, exactly the size of effect most tests exist to detect, which is why bot filtering belongs in test setup.
Uneven distribution is the core problem
If a bot wave hits during a test and lands unevenly, perhaps because one variant's extra script slowed it or a security rule treated the variants differently, the variant with more bots shows worse conversion for reasons nothing to do with its design.
Check the split itself before reading results: variant traffic proportions over time, bounce and session duration distributions, and any hours where one variant received a surge the other did not. A clean test shows boring, even splits.
Filter before analysis, define before launch
Known bot sources, data centre traffic, and non human user agents should be excluded from test data where tooling allows, and the exclusion rules written into the test plan before results arrive. Filtering rules invented after seeing results can be tuned, even unconsciously, to save a preferred variant.
Segment the results as a cross check: returning customers and logged in sessions are overwhelmingly human. If the human heavy segment disagrees with the headline result, the headline is probably measuring bots.
Slow down on small effects
When a test's reported lift is small and the store's bot share is meaningful, treat the result as provisional. Extend the test, rerun it with stricter filtering, or require a larger sample before redesigning a page around the finding.
Keep a record of bot incidents alongside test dates. A test that ran through a scraping wave carries an asterisk forever, and future teams reading the result library deserve to know.
Related reading
If this topic is useful, these earlier guides go deeper on connected questions: Bots Scraping Inventory and Price Feeds: What the Traffic Means, Bot Traffic During Flash Sales and Limited Drops, and Bots and Gift Card Balance Checking Attacks.
Frequently asked questions
Why do bots not cancel out between variants?
Because they do not arrive evenly. Loading speed, scripts, and security rules differ slightly between variants, and bot traffic comes in waves that can coincide with one variant's sample more than the other's.
What is the quickest sign bots bent a test result?
An uneven traffic split, strange bounce or duration patterns in one variant, or a result that reverses when you look only at logged in or returning visitors, who are almost all human.
Should bot filtering be decided before or after a test?
Before. Exclusion rules set in the test plan protect the result from motivated reasoning. Rules chosen after seeing results can be shaped, even honestly, by which variant the team prefers.
What should a store do with a suspect test result?
Treat it as provisional: extend or rerun with stricter filtering, and note the bot incident in the results record. Redesigning pages on bent data costs more than a repeated test.