AB Testing: Why You Shouldn’t Focus on Conversion Rate

When A/B testing web design, copy, merchandising or promotions, the most important thing is understanding what metrics you will use to determine success. It’s not always “conversion rate.”

Conversion rate is typically expressed as number of conversion actions (sales, email sign ups, clicks on calls to action) divided by total visitors or unique visitors.

But focusing on conversion rate without understanding its relationship to other metrics may lead you to wrong conclusions in your testing. Consider the following:

• Price and promotions test: More people bought, but at a lower price – profit did not increase
• Merchandising test: More people bought, but items per sale were lower – profit did not increase
• Merchandising test: More people purchased, but with a lower average order value because cross sells were removed from checkout
• Email offer price, promotions or coupon test: More people purchased, but a percentage would have purchased anyway without the discount
• Pre-checked email opt in test: More people signed up for email, but reported your messages as spam because they signed up unwittingly
• Cart button test: More people initiated checkout, but abandonment the same because the real problem lies in the funnel
• Remove negative reviews test: More people purchased the item, but more tried to refund because item wasn’t explained truthfully, negative reviews suppressed
• Banner ad test: More traffic was driven to your site, conversion rate decreased, revenue increased

A real life example: Skritter boasts a 218% conversion improvement with no impact on sales. It’s landing page tripled the number of clicks to its software demo, but had no impact on sales or revenue.

Beyond conversion rate, are you tracking the following?

• Revenue
• Revenue per visitor
• Average order value
• Items per order
• Gross margin
• Margin per visit (profit)
• Margin per customer
• Revenue less returns
• Return rate
• Repurchase rate
• Lifetime customer value

Many of these metrics don’t come out of the box with your analytics tool. Some of them can be imported into your paid / enterprise analytics tool from other systems (check with your vendor). Yahoo Web Analytics and Omniture have both shared instructions for importing COGS (cost of goods sold), for example. Google Analytics doesn't allow you to import cost data, which is one reason to explore a paid tool (unless you’re a wizard with Excel, then you could export conversion data out of GA).

If you plan on A/B or multivariate testing in 2011, make sure you incorporate one or more financial metrics in your test analysis (at minimum, revenue and revenue per visitor which come out of the box).