Should You Avoid Testing During a Traffic Spike?

You may have heard the advice to avoid A/B and multivariate testing during a traffic spike. This is sound advice for the most part, but I suggest there are times when testing during a traffic spike is exactly what you should do.

Testing at the beginning of a predictable spike can actually help you optimize for sales during the remainder. More traffic means faster test completion, and changes can be applied for the rest of the wave. For example, if you can run test beginning Black Friday that reaches statistical significance in 5 days, you have a few weeks to benefit from the learnings of the test.

The reason why testing during a traffic spike (whether a seasonal spike or due to a new product launch, sales event or promotional campaign) is not advised is customer behavior may change during the busy period. This may introduce bias into your test which can skew your results. But we need to think this through, and ask ourselves which tests could be affected by different user behavior. Will back-to-school shoppers behave differently than regular folks during a checkout test where a button is made more prominent? Or can a test be reasonably applied to all visitors under any circumstance?

A common test variable is the size/color/placement/label of a checkout button. We wouldn't expect holiday shoppers to be more influenced by a red or green button than any other shopper at any other time. The test is measuring the conversion impact for a change in design, not merchandising or promotions - the variables that can truly move the needle during high season and other heavy traffic periods. These are variables you can't reliably test during slower periods - behavior is different.

Here are some examples of seasonal test ideas (hint: they are all related to merchandising, pricing and promotions):

• $/% discounts vs free shipping
• cross sells (placement of) or # of cross sells displayed on product pages
• cross sells in cart vs. not in cart
• cross sell labels (e.g. 'people who bought X also bought Y' vs. 'essential add-ons')
• featured products/categories on home page
• promotional banners - imagery, creative and offers
• gift finder on home page vs. none
• default sort on search/category sort (e.g. sort by relevance, best sellers, top rated, etc, measure click-through)

Tips for testing during high traffic periods

1. Know when to start and stop your test. Don't start before the spike or let it continue beyond
2. Craft your test to match expected behavior during the spike (relevance!)
3. Focus on variables that have the highest potential impact on sales
4. Understand traffic and conversion improvement estimates to ensure your test will finish quickly during the spike (use an estimate calculator)
5. Use a limited number of test recipes so the test finishes quickly
6. Complete other 'general' tests before the seasonal/spike test to reduce bias from other tests (e.g. customer sees free shipping over $50 on home page but encounters a one-page checkout test when ready to buy
7. Communicate with your email and PPC departments to sync up on dates, or exclude email and PPC traffic from seasonal tests)

What about unexpected spikes?

Was your business featured on Oprah Rachel Ray during a test? Did your PPC squad turn up the juice on your paid search campaign? You may want to re-test after traffic returns to normal and compare to the spike to validate the results.

Wanna learn more about site optimization testing? We're teaming up with Marketing Sherpa for our August webinar Putting Your Best Site Forward: Making the Case for Consistent Testing on Enterprise Ecommerce Sites on Wednesday, August 24, 2011 from
9:00 to 9:45am PDT.