AB Testing: Why You Shouldn’t Focus on Conversion Rate

When A/B testing web design, copy, merchandising or promotions, the most important thing is understanding what metrics you will use to determine success. It’s not always “conversion rate.”

Conversion rate is typically expressed as number of conversion actions (sales, email sign ups, clicks on calls to action) divided by total visitors or unique visitors.

But focusing on conversion rate without understanding its relationship to other metrics may lead you to wrong conclusions in your testing. Consider the following:

  • Price and promotions test: More people bought, but at a lower price – profit did not increase
  • Merchandising test: More people bought, but items per sale were lower – profit did not increase
  • Merchandising test: More people purchased, but with a lower average order value because cross sells were removed from checkout
  • Email offer price, promotions or coupon test: More people purchased, but a percentage would have purchased anyway without the discount
  • Pre-checked email opt in test: More people signed up for email, but reported your messages as spam because they signed up unwittingly
  • Cart button test: More people initiated checkout, but abandonment the same because the real problem lies in the funnel
  • Remove negative reviews test: More people purchased the item, but more tried to refund because item wasn’t explained truthfully, negative reviews suppressed
  • Banner ad test: More traffic was driven to your site, conversion rate decreased, revenue increased

A real life example: Skritter boasts a 218% conversion improvement with no impact on sales. It’s landing page tripled the number of clicks to its software demo, but had no impact on sales or revenue.

Beyond conversion rate, are you tracking the following?

  • Revenue
  • Revenue per visitor
  • Average order value
  • Items per order
  • Gross margin
  • Margin per visit (profit)
  • Margin per customer
  • Revenue less returns
  • Return rate
  • Repurchase rate
  • Lifetime customer value

Many of these metrics don’t come out of the box with your analytics tool. Some of them can be imported into your paid / enterprise analytics tool from other systems (check with your vendor). Yahoo Web Analytics and Omniture have both shared instructions for importing COGS (cost of goods sold), for example. Google Analytics doesn’t allow you to import cost data, which is one reason to explore a paid tool (unless you’re a wizard with Excel, then you could export conversion data out of GA).

If you plan on A/B or multivariate testing in 2011, make sure you incorporate one or more financial metrics in your test analysis (at minimum, revenue and revenue per visitor which come out of the box).


Related Articles

10 Responses to “AB Testing: Why You Shouldn’t Focus on Conversion Rate”

  1. I agree that focusing on the headline conversion rate would be misleading unless the objective was simply to test if a lower price, as in first example, did increase overall purchases.

    I would assume, however, that any serious marketer would be attempting to extract some more enlightening data eg to test price elasticity and identify a price point that found the equilibrium point at which to test other factors to increase the overall contribution to bottomline.

    Although GA doesn’t yet enable COGS it still provides the ability to identify, for ecommerce sites, avg value contribution and other sales related metrics by channel, referrer, visitor type etc through the custom reporting facility which, admittedly, is not as informative but a relatively good starting point in identifying some of the factors influencing site visitors’ behaviour.

  2. You make some good points but we have found conversion rates are always the best thing to consider … you just need to adjust the outcome you consider a successful conversion.

  3. I would also recommend considering payment types and usage. Transaction fees can vary by payment type, even within payment types if connected with programs such as Rewards/Loyalty. The behaviors of customers can vary by payment type as well, impacting conversion, revenue and margin.

  4. You must also consider Cost of Acquisition of a Customer i.e. how much did it cost to get that customer and how much do they spend with you over a 12 month cycle. It may be good to take a lower profit on promotions if you can establish a long term customer who buys repeatedly.

  5. Different departments will always have different KPIs. As conversion rate is most directly linked to the bottom line, it’ll be the main one; others will consider bounce rate, time on site, click-throughs etc, depending on their own focus.

  6. You must also consider Cost of Acquisition of a Customer i.e. how much did it cost to get that customer and how much do they spend with you over a 12 month cycle. It may be good to take a lower profit on promotions if you can establish a long term customer who buys repeatedly. this could really be something that people would like to see and read.

  7. It is importnant to know that when your CR is high it doesn’t mean anyway that your profit is high. The article explains this very well. I also thing as Anthony regarding the rewards/Lotalty!

  8. this is a much needed wake up for retailers. I can’t say how many a/b tests I’ve run where GWO tells me I have a winner, but when I look at actual dollars I don’t. Google doesn’t let you track average order values thru GWO (that I’m aware of) and that’s a shame.

  9. Bruno says:

    Bull… I bet she’s never heard of “Ceteris paribus”.

  10. Alvin Tan says:

    Conversion rate is nonetheless useful in measuring how well a particular change or campaign performs in relation to the conversion goal. No doubt, if we are tracking sales or profits, conversion rates alone may not be useful, because it does not capture the dollar amount of sales generated or customer acquisition costs. However, if our goal is to measure how many people eventually proceed to a certain next stage of the funnel, conversion rate is useful. The reason that Skritter managed to increase their conversion rate by 218% is that 1) their conversion rate was small to begin with (it’s easy to increase rates by 200% when their original rate is 1% or whatever) and 2) they merely succeeded in moving the bottleneck from one stage to another.

Leave a Reply

© 2014 Get Elastic Ecommerce Blog. All rights reserved. Site Admin · Entries RSS · Comments RSS