Free or Fee: What’s The Diff Between GWO and Paid Testing Tools?

You may have heard there’s “no excuse for not testing” thanks to the incredible free gift Google hath bestowed upon us, Google Website Optimizer. While this is true, GWO is a powerful tool at an attractive “price” - what about paid tools? How do you know whether free is good enough, or if you need a little somethin’ more?

There are at least 17 web optimization tools available, according to WhichMVT.com. WhichMVT.com has built a fabulous resource to compare the ins and outs of the player landscape, including actual users’ reviews. (If you have experience with a tool, why not give back to the community and submit your own review)?

I won’t rehash the information on WhichMVT. Rather, today let’s look at the extremes between Google Website Optimizer (the only free tool available) and Adobe Omniture Test and Target. TnT is not necessarily the most expensive option, but one of the more popular tools among enterprises.

Google Website Optimizer

Free

It’s tough to argue with free, even if a tool has less functionality than paid tools. GWO is suitable for the novice tester, as too many features and functions can be confusing and can actually overcomplicate your testing. But for larger enterprises and seasoned testers, the price of free may be too high.

Full factorial only

A full factorial test refers to multivariate testing only. It means that your test must include every permutation of your variables. For example, if you are testing 4 headlines, 4 hero shots and 4 call to action colors, your full factorial is 4x4x4, or 64 recipes. That requires a lot of traffic to be able to complete the test with statistical validity in a reasonable time frame. GWO does not support the Taguchi Method, which allows you to exclude certain recipes from the experiment. Google’s full-factorial restriction is not necessarily a negative. Using only a portion of total recipe permutations does allow you to finish tests faster, but also leaves data on the table - you’ll never know if the excluded recipes were actually winners.

Limited segmentation/personalization capability

With GWO, it’s all-or-nothing. Though you may choose to include less than 100% of visitors in your experiment, you cannot specify which visitors will be included - it will be randomized. Segmentation can be helpful when you want to gradually “roll out” new tests, like a radical site redesign to only new visitors at first. Existing visitors may have learned the processes of your existing design, and the new one would actually reduce goal conversion because it requires new learning on behalf of the user. This is the approach Amazon uses when rolling out major changes. It begins first with new, uncookied visitors, then tests with a sample of returning visitors, and then does a full roll-out after a change has proven successful.

Equal traffic to recipes

Another drawback is GWO will serve test pages equally to the percentage of traffic you specify for your experiment. If you have a strong performing page that you want to make even stronger, you actually risk losing money during the test period if your challenger recipes are not strong performers. Sending only a portion of traffic to test recipes, for example, can reduce this risk. For example, you may choose to send 70% to your control and 30% to your challenger, or 50% to your control and 25% to your two challengers.

Further, if you decide to include only 50% in your experiment, you will reduce risk (25% of traffic to control, 25% to challenger). The remaining 50% will see your control, but you will not collect comparative data. Your test will take longer to reach statistical significance, and you won’t have as much confidence in the results as you would if you could have used 100% of visitors in the experiment, but sent 75% to your control and 25% to your experiment.

Limits on variables/branching factors, recipes

Google caps the number of recipes, variables and branching factors (variations of the variables, like red, orange, green and blue for a button color) to 10,000 - which is more than enough even for multivariate testing. Larger enterprises and seasoned testers *may* find this too restrictive.

Can’t import data

GWO is tied to Google Analytics, which cannot take data like cost of goods sold into account. Remember, it’s not just conversion rate that matters, but what improves your bottom line. It’s possible a test may appear to be a winner but is actually less profitable. For example, conversion rate may skyrocket with a free shipping promotion, but you may be losing more money than it's worth.

Omniture Test&Target

Expensive

Premium functionality comes with a premium price - typically in the 4-figures-a-month range. If you’re not just tire-kicking and ongoing site testing is critical to your enterprise, Test and Target may pay for itself.

But many enterprises who already use Omniture’s Site Catalyst analytics tool may want to start testing, but can’t budget for Test and Target. Never fear! Google Website Optimizer data can be imported to enterprise analytics tools. Here’s a tutorial for exporting it to Site Catalyst.

Plays nicely with other products in Omniture suite

While you can integrate GWO with Site Catalyst, you’re likely to have an easier time with products of the same family if you already use other Omniture products. Data can be shared across the suite. Terminology is consistent. Technology vendors tend to create their own terminology to describe the same things. Omniture Site Catalyst, for example, uses "eVars" to describe custom variables, while Google Analytics uses "Events." Web analytics tools may also measure KPIs like “unique visitors” and “conversion rate” slightly different. You’re also likely to get more technical support from a company you’re spending thousands with than a company with hundreds of thousands of free users.

Predictive learning

Test and Target allows a degree of personalization by building visitor profiles of anonymous users based on site behavior. It also accepts offline data like a customer’s credit score or time on file, which may be helpful for personalization.

Other goodies

Omniture (as well as other paid tools) provide you the ability to run multiple tests concurrently, segment traffic, control the percentage of traffic sent to the control vs. treatment recipes and the to scale beyond 10K branching factors.

Should you buy a tool?

If you haven’t bought into a paid tool and you’re just getting your feet wet with testing, there’s nothing wrong with Google Website Optimizer to, as Bryan Eisenberg says, “get good at free, then look at paid tools.”

Only serious testers who are actually going to take advantage of the additional functions of enterprise optimization tools should consider paying thousands per month. This is not for the casual tester! Even so, more important than the tool are the brains behind your testing program. Remember the 90/10 rule? You should spend 10% on tools and 90% on people. Starting off with a lesser-priced tool to build the experience and to demonstrate an ROI for your testing program is more important than having the Cadillac in the garage.

And remember, there’s more affordable paid tools out there. WhichMVT is a great resource that can help you differentiate the vendors and find a tool that fits both your needs and your budget.

Interested in learning more about A/B and multivariate testing? I’ll be presenting a webinar with Hubspot on June 9 at 4pm EST titled Lose Your Gut (Feel) in 60 Minutes: Site Optimization Testing Boot Camp. Sign up today!