Comments on: A/B Test Case Study: Can Split Test Results Be Trusted?

By: Tariq

Tariq — Thu, 05 Aug 2010 20:19:53 +0000

Re: A/A testing

Did a similar experiment with troubling results…

http://www.google.com/support/forum/p/websiteoptimizer/thread?fid=301a7fae4728adb900048b4638bcc760&hl=en

By: Matt Nolker

Matt Nolker — Thu, 05 Aug 2010 13:37:11 +0000

Evan Miller published a story entitled “How to Not Run an A/B Test.” Might be worth a quick read. He demonstrates that the size of your sample can influence results if you don’t freeze it in advance at a set number. http://www.evanmiller.org/how-not-to-run-an-ab-test.html

By: Robert Kingston

Robert Kingston — Wed, 21 Jul 2010 09:57:56 +0000

Thanks Linda… That’s what I’ve been using at work too.

By: Linda Bustos

Linda Bustos — Tue, 20 Jul 2010 23:56:08 +0000

(Test $AOV – Control $AOV) / Control $AOV

By: Rob Kingston

Rob Kingston — Mon, 19 Jul 2010 03:57:07 +0000

How did you calculate the increase in average order value?

By: Ian Clarke

Ian Clarke — Fri, 05 Mar 2010 02:12:13 +0000

Let’s say you had 1000 impressions, and 200 conversions – meaning a 20% conversion rate. But how certain can you be of this conversion rate given how many impressions it is based on? Fortunately, math can help – what you need to use is something called the “beta distribution”. The beta distribution takes 2 parameters, A and B. In your case, A is 200+1 or 201, and B is 1000-200+1 or 801. Now we can determine the accuracy of our 20% conversion rate estimate. The variance is A*B/((A+B)^2 * (A+B+1)), which in this case is 201*801/((201+801)^2*(201+801+1)) which is 0.000159879285.

The standard deviation is the square root of the variance, which in this example case is 0.0126443381 or 1.3%. If we double the standard deviation, we get our 95% confidence interval, which is from 20%-(1.3%*2) to 20%+(1.3%*2).

It means that based on the fact that we had 200 conversions with 1000 impressions, there is a 95% chance that the real conversion rate is between 17.4% and 22.6%.

Using this approach, you can take the guess work out of deciding how much data is enough to make a decision. This type of thing is the bread and butter of our company, SenseArray.

By: Janis Lanka

Janis Lanka — Fri, 05 Mar 2010 00:22:03 +0000

@Will – you have a very good point and 5% mark is a good practice. And b/c of “margin for error” we are looking at additional data (Conversion Rate, AOV, etc.) to make most educated decisions.

By: Will Reinhardt

Will Reinhardt — Thu, 04 Mar 2010 22:10:39 +0000

Your 4.97% variance is a good explanation why there needs to be a threshold when choosing a winner. We typically recommend at least a 5% difference for this very reason. “Margin for error” doesn’t always mean you made a mistake, it can simply mean that you’re allowing for the differences in your audience.

By: Rex Dixon

Rex Dixon — Thu, 04 Mar 2010 21:56:26 +0000

Please share these results on our A/B Tests site.

By: John Town

John Town — Wed, 03 Mar 2010 23:25:52 +0000

Really depends on what your trying to split test – Your going to need enough data there to analyze and make a good decision – Being a statistician in this case is very helpful or using a high end program to do this.

Split testing is not something you just eyeball.