A/B Case Study: How MEC Found a 404 Page Winner

Last year, Mountain Equipment Co-op (MEC) shared its case study on 404 “Not Found” pages with Anne Holland of Which Test Won. This test won a Gold Ribbon in the Ecommerce category in the 2009 Testing Awards.

If you missed it, go on and “test your gut” – see if you can guess which design converted best.

If you guessed the 404 page with the image, you are correct. It achieved not only a 3.56% lift but also a 73.2% boost in revenue per visitor.

What you didn’t see on Which Test Won were the other versions tested by Amadesa, the conversion optimization firm that conducted the experiments for MEC. Check ‘em out:

Control

Click image to enlarge

V1.00 – Text only

  • Updated text – messaging why a user might have reached the 404 page
  • Conversion rate of 37%

Click image to enlarge

V1.01 – Text ,Image, & Shortened Links

  • Updated text – messaging why a user might have reached the 404 page
  • Addition of image
  • Addition of search
  • Shortened link list
  • Conversion rate of 33.74%
  • Revenue per visit lift of -29.12%

Click image to enlarge

V1.02 – Image, text, & shortened links

  • Updated text – messaging why a user might have reached the 404 page
  • Addition of image
  • Addition of search
  • Shortened link list
  • Conversion rate of 37.74%
  • Revenue per visit lift of 38.44%

Click image to enlarge

V1.03 – Large image, text, & dropdown

  • Updated Text – messaging why a user might have reached the 404 page
  • Addition of image
  • Addition of search
  • Product dropdown
  • Conversion rate of 38.32%
  • Conversion lift of 3.56%
  • Revenue per visit lift of 73.2%

Click image to enlarge

The winner

The winning version had a conversion rate of 38.32% and a conversion lift of 3.56%. Overall, this version presented a revenue per visit lift of 73.2%.

MEC found that providing users with an abundance of links (in the first version) did not engage the user, rather overwhelmed them to the point of abandonment. The winning version provided readers with an apology for the error, as well as more appealing imagery and color scheme that better fit the lifestyle of MEC customers. It also provided a search field and drop box that simplified the re-navigation process for users – a successful vehicle for re-engaging.


Related Articles

10 Responses to “A/B Case Study: How MEC Found a 404 Page Winner”

  1. Great article I took a picture and am going to send it off to my designer to incorporate into my website.

  2. pk says:

    Cool results and all but isn’t the goal for the user not to get to a 404 page? IMO the effort that went to optimizing a page should have went to fixing the root cause that lead the user to a 404 page.

  3. I’m just a little confused by this article. So the winning (or best) version was “V1.03 – Large image, text, & dropdown”? The text for that version says:
    # Conversion lift of
    # Revenue per visit lift of 14.83%
    While the text below about the winning version says “…a conversion lift of 3.56%. Overall, this version presented a revenue per visit lift of 73.2%.”

    And if that’s the best version, how come their site is using what appears to be V1.01?

    Definitely some interesting results and in line with what people expect from websites these days with regard to usability and appearance. The control version and the text-only version (V1.00) definitely have the feel of pages from the early days of the web.

  4. Testing the 404 page is a great idea in theory, but it may not be worth doing for most sites because the traffic to the 404 page will be too small. I looked at the web analytics data for an e-commerce site that is multiple times bigger than mec.ca and they only get 1200 unique pageviews of the 404 page per month.

    One thing I don’t understand are the high conversion rates, e.g 37.74% and 38.32%. What is a conversion in this context? Is this people clicking on any link on the 404 page? Did Amadesa actually confirm that they got statistically valid results?

  5. @gaspoweredcars Why not split test the new version against your existing one to see if the new version has a positive effect? There is danger in taking what works for one site/customer and applying it to your own without testing. If you send all the traffic to the new version, you can’t quantify the lift.

    @pk sometimes 404 pages are inevitable, think of someone linking to your site with a typo in the URL. Your server will not find the correct URL itself without some redirection on your side. Having a 404 page, you can see what URL referrals are causing “not founds” and contact these sites to update their links. In the meantime, you want to keep customers engaged on your site, and not have them abandon. I know other sites will simply redirect any 404 to the home page – but that’s not the best user experience, either.

    @David Bennet
    I’ve updated the article to correct the blank and the revenue per visit metric, which indeed is 73.2%, not 14.83. I can’t speak to why the current version on Mec.ca differs from the winner of this test. Could be another test.

    @Michael Whitaker
    The KPIs were bounce rate, conversion and Revenue per Visitor. I will double check with Amadesa about validity, thanks.

  6. Bruce says:

    Testing the 404 page is a great idea in theory, but it may not be worth doing for most sites because the traffic to the 404 page will be too small. I looked at the web analytics data for an e-commerce site that is multiple times bigger than mec.ca and they only get 1200 unique pageviews of the 404 page per month.

    One thing I don’t understand are the high conversion rates, e.g 37.74% and 38.32%. What is a conversion in this context? Is this people clicking on any link on the 404 page? Did Amadesa actually confirm that they got statistically valid results?

  7. I verified with Amadesa re: the validity/conversion rate:

    · MEC found from their website analytics data that there was significant traffic and a high bounce and exit rate on this 404 page

    · They tracked RPV for visitors who reached this page for each test version via Amadesa java script tags on the 404 page, checkout, and checkout confirmation pages

    · Over the course of the test we achieved 95% confidence for conversion rate lift

    · Conversion was defined as users successfully getting to a product detail page or completing a purchase

  8. Nick Piercy says:

    Why, exactly three months later, is this back to the control version if the improved version is so much better?

  9. tom says:

    So, did they actually purposefully send traffic to the 404 page? Or just rely on normal levels of traffic?

Leave a Reply

© 2014 Get Elastic Ecommerce Blog. All rights reserved. Site Admin · Entries RSS · Comments RSS