Comments on: A/B Case Study: How MEC Found a 404 Page Winner http://www.getelastic.com/mec-ab-test/ #1 Subscribed Ecommerce Blog Tue, 07 Feb 2012 15:39:47 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Linda Bustoshttp://www.getelastic.com/mec-ab-test/comment-page-1/#comment-64422 Linda Bustos Mon, 14 Nov 2011 21:47:05 +0000 http://www.getelastic.com/?p=7848#comment-64422 Normal levels of traffic :) Normal levels of traffic :)

]]> By: tomhttp://www.getelastic.com/mec-ab-test/comment-page-1/#comment-64421 tom Thu, 10 Nov 2011 19:56:23 +0000 http://www.getelastic.com/?p=7848#comment-64421 So, did they actually purposefully send traffic to the 404 page? Or just rely on normal levels of traffic? So, did they actually purposefully send traffic to the 404 page? Or just rely on normal levels of traffic?

]]> By: Nick Piercyhttp://www.getelastic.com/mec-ab-test/comment-page-1/#comment-27136 Nick Piercy Mon, 16 Aug 2010 10:46:00 +0000 http://www.getelastic.com/?p=7848#comment-27136 Why, exactly three months later, is this back to the control version if the improved version is so much better? Why, exactly three months later, is this back to the control version if the improved version is so much better?

]]> By: Linda Bustoshttp://www.getelastic.com/mec-ab-test/comment-page-1/#comment-23014 Linda Bustos Fri, 21 May 2010 21:43:15 +0000 http://www.getelastic.com/?p=7848#comment-23014 I verified with Amadesa re: the validity/conversion rate: · MEC found from their website analytics data that there was significant traffic and a high bounce and exit rate on this 404 page · They tracked RPV for visitors who reached this page for each test version via Amadesa java script tags on the 404 page, checkout, and checkout confirmation pages · Over the course of the test we achieved 95% confidence for conversion rate lift · Conversion was defined as users successfully getting to a product detail page or completing a purchase I verified with Amadesa re: the validity/conversion rate:

· MEC found from their website analytics data that there was significant traffic and a high bounce and exit rate on this 404 page

· They tracked RPV for visitors who reached this page for each test version via Amadesa java script tags on the 404 page, checkout, and checkout confirmation pages

· Over the course of the test we achieved 95% confidence for conversion rate lift

· Conversion was defined as users successfully getting to a product detail page or completing a purchase

]]> By: Brucehttp://www.getelastic.com/mec-ab-test/comment-page-1/#comment-22966 Bruce Wed, 19 May 2010 09:24:10 +0000 http://www.getelastic.com/?p=7848#comment-22966 Testing the 404 page is a great idea in theory, but it may not be worth doing for most sites because the traffic to the 404 page will be too small. I looked at the web analytics data for an e-commerce site that is multiple times bigger than mec.ca and they only get 1200 unique pageviews of the 404 page per month. One thing I don't understand are the high conversion rates, e.g 37.74% and 38.32%. What is a conversion in this context? Is this people clicking on any link on the 404 page? Did Amadesa actually confirm that they got statistically valid results? Testing the 404 page is a great idea in theory, but it may not be worth doing for most sites because the traffic to the 404 page will be too small. I looked at the web analytics data for an e-commerce site that is multiple times bigger than mec.ca and they only get 1200 unique pageviews of the 404 page per month.

One thing I don’t understand are the high conversion rates, e.g 37.74% and 38.32%. What is a conversion in this context? Is this people clicking on any link on the 404 page? Did Amadesa actually confirm that they got statistically valid results?

]]> By: Linda Bustoshttp://www.getelastic.com/mec-ab-test/comment-page-1/#comment-22938 Linda Bustos Tue, 18 May 2010 15:24:33 +0000 http://www.getelastic.com/?p=7848#comment-22938 @gaspoweredcars Why not split test the new version against your existing one to see if the new version has a positive effect? There is danger in taking what works for one site/customer and applying it to your own without testing. If you send all the traffic to the new version, you can't quantify the lift. @pk sometimes 404 pages are inevitable, think of someone linking to your site with a typo in the URL. Your server will not find the correct URL itself without some redirection on your side. Having a 404 page, you can see what URL referrals are causing "not founds" and contact these sites to update their links. In the meantime, you want to keep customers engaged on your site, and not have them abandon. I know other sites will simply redirect any 404 to the home page - but that's not the best user experience, either. @David Bennet I've updated the article to correct the blank and the revenue per visit metric, which indeed is 73.2%, not 14.83. I can't speak to why the current version on Mec.ca differs from the winner of this test. Could be another test. @Michael Whitaker The KPIs were bounce rate, conversion and Revenue per Visitor. I will double check with Amadesa about validity, thanks. @gaspoweredcars Why not split test the new version against your existing one to see if the new version has a positive effect? There is danger in taking what works for one site/customer and applying it to your own without testing. If you send all the traffic to the new version, you can’t quantify the lift.

@pk sometimes 404 pages are inevitable, think of someone linking to your site with a typo in the URL. Your server will not find the correct URL itself without some redirection on your side. Having a 404 page, you can see what URL referrals are causing “not founds” and contact these sites to update their links. In the meantime, you want to keep customers engaged on your site, and not have them abandon. I know other sites will simply redirect any 404 to the home page – but that’s not the best user experience, either.

@David Bennet
I’ve updated the article to correct the blank and the revenue per visit metric, which indeed is 73.2%, not 14.83. I can’t speak to why the current version on Mec.ca differs from the winner of this test. Could be another test.

@Michael Whitaker
The KPIs were bounce rate, conversion and Revenue per Visitor. I will double check with Amadesa about validity, thanks.

]]> By: Michael Whitakerhttp://www.getelastic.com/mec-ab-test/comment-page-1/#comment-22905 Michael Whitaker Mon, 17 May 2010 17:14:44 +0000 http://www.getelastic.com/?p=7848#comment-22905 Testing the 404 page is a great idea in theory, but it may not be worth doing for most sites because the traffic to the 404 page will be too small. I looked at the web analytics data for an e-commerce site that is multiple times bigger than mec.ca and they only get 1200 unique pageviews of the 404 page per month. One thing I don't understand are the high conversion rates, e.g 37.74% and 38.32%. What is a conversion in this context? Is this people clicking on any link on the 404 page? Did Amadesa actually confirm that they got statistically valid results? Testing the 404 page is a great idea in theory, but it may not be worth doing for most sites because the traffic to the 404 page will be too small. I looked at the web analytics data for an e-commerce site that is multiple times bigger than mec.ca and they only get 1200 unique pageviews of the 404 page per month.

One thing I don’t understand are the high conversion rates, e.g 37.74% and 38.32%. What is a conversion in this context? Is this people clicking on any link on the 404 page? Did Amadesa actually confirm that they got statistically valid results?

]]> By: David Bennetthttp://www.getelastic.com/mec-ab-test/comment-page-1/#comment-22904 David Bennett Mon, 17 May 2010 17:02:57 +0000 http://www.getelastic.com/?p=7848#comment-22904 I'm just a little confused by this article. So the winning (or best) version was "V1.03 – Large image, text, & dropdown"? The text for that version says: # Conversion lift of # Revenue per visit lift of 14.83% While the text below about the winning version says "...a conversion lift of 3.56%. Overall, this version presented a revenue per visit lift of 73.2%." And if that's the best version, how come their site is using what appears to be V1.01? Definitely some interesting results and in line with what people expect from websites these days with regard to usability and appearance. The control version and the text-only version (V1.00) definitely have the feel of pages from the early days of the web. I’m just a little confused by this article. So the winning (or best) version was “V1.03 – Large image, text, & dropdown”? The text for that version says:
# Conversion lift of
# Revenue per visit lift of 14.83%
While the text below about the winning version says “…a conversion lift of 3.56%. Overall, this version presented a revenue per visit lift of 73.2%.”

And if that’s the best version, how come their site is using what appears to be V1.01?

Definitely some interesting results and in line with what people expect from websites these days with regard to usability and appearance. The control version and the text-only version (V1.00) definitely have the feel of pages from the early days of the web.

]]> By: pkhttp://www.getelastic.com/mec-ab-test/comment-page-1/#comment-22902 pk Mon, 17 May 2010 14:55:27 +0000 http://www.getelastic.com/?p=7848#comment-22902 Cool results and all but isn't the goal for the user not to get to a 404 page? IMO the effort that went to optimizing a page should have went to fixing the root cause that lead the user to a 404 page. Cool results and all but isn’t the goal for the user not to get to a 404 page? IMO the effort that went to optimizing a page should have went to fixing the root cause that lead the user to a 404 page.

]]> By: gaspoweredrccarshttp://www.getelastic.com/mec-ab-test/comment-page-1/#comment-22901 gaspoweredrccars Mon, 17 May 2010 13:27:26 +0000 http://www.getelastic.com/?p=7848#comment-22901 Great article I took a picture and am going to send it off to my designer to incorporate into my website. Great article I took a picture and am going to send it off to my designer to incorporate into my website.

]]>