A/B Test Case Study: Location of Size and Color Options Mattered

This post is contributed by Janis Lanka (@janislanka, who manages front-end development for Elastic Path Software.

Continuing our case study in conversion optimization for the Official Vancouver 2010 Olympic Store (see last week's post if you missed it), today we'll share the results of our product detail page testing.

We began with potential issues with the current design:

• Color and size selection were provided in drop-down menus which were difficult to notice and understand. For example, what do the colors "Juniper", "Pacific" or "Fern" really look like?
• The Add to Cart button and price often fell below the fold and were hard to find if a product had a long description.

Considering above, the following tweaks (Variation A) were introduced:

• Most crucial elements such as Add to Cart button, price, and amount selector were moved to the right in their own column and switch places with Shop With Confidence block
• Drop-downs were replaced with visual selectors
• Product review information was moved under the product title
• Alternative image thumbnails were increased in size

During the initial review of Variation B, we gathered feedback suggesting that visual selectors for color need to be closer to the main product photo. The reason being some on our team felt the color selectors were too distant from the product photo. We wanted to test this as an alternative hypothesis. Thus, location of visual selectors was the only difference between Variation A and Variation B.

Control

Variation A

Variation B

What We Learned

During the experiment, all three variations were equally distributed to 100% of all traffic. 15 days and 2567 conversions later, Google Website Optimizer found a high confidence winner. Consensus was that both new variations would perform better than the control one, but variation B was the ultimate winner:

• Variation B converted (GWO) 19.9% better than the control variation
• Variation A converted (GWO) 11.2% better than the control variation

After reviewing this test, some suggested that Variation A performed worse because most products had rather long "short" description, pushing those selectors fairly close or below the fold. A good follow-up test by splitting description into short and long would answer this.

Finally, it is difficult to say what change exactly contributed the most - the move of Add to Cart button or the change of drop-downs into visual selectors. Unfortunately, we didn't have the luxury of time to localize these experiments and provide more detailed report on that. However, we can say with confidence that fairly minor UI changes can result in rather high improvement.