AB Testing Case Study: The Dangers of Overreacting
This is a guest post / case study contributed by Dr. Pete Meyers of User Effect.
When economic times are tough, even the most level-headed of us sometimes make the mistake of changing our websites just for the sake of change. If the money’s not coming in, something must be broken, right? Maybe, but without knowing why it’s broken, you’ve got a good a chance of breaking it even more. Consider this post a parable in the dangers of overreacting.
Step 1. The Imaginary Problem
Recently, I was working with a client in the event industry who was concerned about the bounce rate on a key page and the strength of their call to action. To make a long story short, we had a page with multiple buttons, calling users to 3 different actions based on their preference. Those buttons looked something like this (altered slightly for client anonymity):
The client felt that we simply weren’t creating enough urgency with the relatively weak “Enroll Online” call to action. There was a certain logic to that argument, and it fit some conventional wisdom on conversion tactics. As usual, the biggest mistakes often have a certain logic to them.
Step 2. The Ill-advised Solution
So, we set about creating 3 variations on the first button. The logic was that “Enroll Online” was too vague and wasn’t conveying a sense of urgency, so we created a version with “Now” and the more emphatic “Now!” Exclamation points are always a wild-card in calls to action, in my experience, so it seemed worth testing. The new buttons (versions A, B and C), looked like this:
Step 3. The (almost) Disastrous Results
Fortunately, we ran a split-test (A/B/C) through Google Website Optimizer and saw a disturbing pattern very quickly. The most emphatic option (“Enroll Now!”) was showing a 37% drop in conversion! Although version B eventually leveled out a bit and only showed a minor loss, both “Now” options performed worse than the original.
What Happened?
Interpretation is always the hardest part of testing, but I think the story goes something like this. In our rush to make a change, we forgot something very important: most of our visitors require multiple visits to convert. They typically have to compare events, get budget approval, etc., and tend to come back a handful of times before enrolling. By pushing them too hard to enroll Now!, we ultimately forgot who our customers were.
Lessons in Overreacting
Of course the lesson here is an old one: “If it ain’t broke, don’t fix it”. We didn’t have the evidence to suggest our buttons were at fault, and in the rush to change the site, we fixed something that should have been left alone. The second lesson is that testing might just save your life, or at least your ROI. Ultimately, this negative change only went out to a third of our visitors for a couple of weeks. Had we rolled the change out without testing and run it for a few months, that 37% conversion drop would have easily cost my client thousands of dollars.
Dr. Peter J. Meyers is the President of User Effect, a former start-up executive, cognitive psychologist, and (nearly) lifelong programmer. For the past 11 years, Dr. Pete has worked directly with business owners and executives to improve their online return on investment. He has recently published a new e-book: Converting The Believers, a guide to using analytics, usability and testing to drive online sales.
Like This Article?
|
Get New Posts Delivered to You
|
Add to del.icio.us
Sphinn It!
Stumble It
As a consumer I do not want to feel pressured into signing up especially online. Good thing you tested the variations. It is the most important ROI exercise when it comes to online advertising.
I’m not so sure.
I would propose a new variation: the difficult-to-disobey “Enroll Now, Damn You!”
Or, “Cool People Enroll Straight Away by Clicking Here.”
Kidding aside, testing is crucial - and this example demonstrates why precisely. In a hurry to make improvements we often shoot ourselves in the feet. Testing before implementing is the only true safeguard.
Hype… Enroll Now! People “feel” the scarcity. I’m astonished at the Internet Marketing strategies and tactics and the people that still buy into them. Any legitimate business seeking to sell online will often do better treating their audience as the educated people they are. People don’t even know why they react anymore, just that they react. Marketing tactics and outcomes create experiences. How many Bad Experiences are represented by cheesy marketing tactics and buttons that create that kind of urgency hype?
Another reason why there might have been a significant drop is the fact that their users were used to seeing “Enroll Online”. There is the period where users need to get used to the new copy/functionality. And while the functionality/copy might be better (not in this case), it would show worse performance results in short-term. Unfortunately, enough times we make quick decisions based on short-term results…
“Enroll Now!” smacks too much of a cheap sales scam. Almost expect to see something like “Offer only valid for first 50 customers”. It makes me question the value of the item on offer - if I wait and think about it then I’ll realize I don’t want to enroll.
We don’t know the actual initial bounce rate on the page in question. Maybe it was a scary number but typical for the market. And yes, presumably the page also contained sales information regarding the event, so realistic to expect repeat visits.
An interesting case study.
Whatever the cause may be, it can’t necessarily be chalked up to oversight in the midst of a rush. The true value of testing is seen in the vantage point that one would not otherwise have.
Without the test, that multiple visit conversion aspect wouldn’t have come to light as easily.
I agree with Chris, I don’t consider this an overreaction at all though. It was a fair question to ask, and as usual, testing ultimately proved which button worked the best and in the end provided a level of insight about your customer that can certainly be logic applied to future forms/buttons.
Interesting study - I’ve worked with organisations that make “minor” changes like the one in the case as a reaction to a senior exec’s demands. I cringe as even the smallest changes can impact your KPIs and return on ££ - and this study talks to this well. I would be interested in hearing how other organisations are structured to control who fundamentally has the last call on what changes are made on the site. Currently where I work, it is pretty much a free for all. Thanks
Thanks, everyone, for all of the great comments.
Regarding the “overreaction”, I guess I didn’t really get into the back-story, but in this particular case, we did make a reactionary decision based on concerns about falling sales. It’s certainly possible that, under different circumstances, the same decision and test may have made sense.
It’s funny that, while some people thought this was a perfectly natural concept to test, some thought it was overblown and marketing-heavy. Yet another argument for testing - our personal reactions often have very little bearing on our customers’ reality. I’ve seen a strong call to action and what may seem like over-the-top urgency be very effective, and I’ve seen it backfire. Sometimes, you have to try it and see.
@Jon - Funny you should mention bounce rate, as a high bounce rate is one reason we keep testing changes to this particular page/step. Interestingly, almost all of those changes have either been neutral or negative. This step in the process just happens to be a natural one for people to exit on (especially those who need to get an approval, etc.), and it’s very hard to separate out those “natural” bounces from lost customers.
Adding urgency can often work for impulse purchases, but can appear downright silly for carefully-considered (as in most B2B) transactions:
“Find out more about our inventory-control software. Quick, ACT NOW!!”
It’s never a bad idea to test though.