Testing the Untestable: An SEO Title Tag Experiment

Words matter. It’s proven that your choice of verbiage can dramatically impact your email open rates, paid search click through rates and landing page conversions. Marketers spend thousands of hours and dollars testing and tweaking text to find out what performs the best. It’s easy with the wealth of testing tools available for your campaigns, but what about organic SEO? Title tags are not just important for ranking, but also for click through. We can expect the majority of searchers to click on the top result, and possibly top few results when searching for information. But for commercial searches, it takes a bit more effort to figure select a search result that’s relevant to the search intent (i.e. an ecommerce site, not a Wikipedia or blog article) and attractive (a familiar domain, a reasonable offer). Your title tag, like your PPC headline, is your small space to shine. But unfortunately, we’re left to make a gut-feel decision on what title tag is best.

Testing title tags: the problems

SERPs (search engine result pages) introduce many uncontrollable variables, such as Google’s penchant for showing different flavors of results, in different areas on the page, for different keyword searches. For example, a search for ‘riding boots’ shows Google Product Search results that can divert a customer from organic listings, star ratings that draw attention to search ads, and brand links to help users narrow their searches.

Notice how the singular “riding boot” and plural “riding boots” show the same page results, but slightly different related Stores results, and omits Product Search results.

If you’re really observant, you’ll notice the related Stores options are different too. Testing is difficult because you have no way of knowing how often these elements are shown or how they may be skewing your results. Not only that, but search positions are always in flux, and vary depending on the exactkeyword phrase queried, the user’s geolocation, whether the user is logged into a Google account (personalized to search history), the “freshness” of results at any given time, and perhaps even involves estimated page load speed and activities of one’s social graph. And did you know that search engines sometimes override your HTML and create their own title for you based on content on your page or what it thinks your page is about? Finally, the biggest hurdle is that even with Google’s own Website Optimizer tool, you cannot tell Google to show one title tag 50% of the time and another the rest. Google will only index your control page. Test versions are served after the referral, if applicable. This makes A/B testing individual pages’ title tags impossible. The only testing you *could* do is in aggregate.

Testing title tags: the workaround

Get Elastic reader Sander Daniels' team at Thumbtack.com found a workaround to A/B testing title tags in aggregate. Using a home-grown testing platform in its ecommerce back end, the test ran for 7 days, Thumbtack split its URLs into 3 buckets, each containing thousands of URLs. The buckets consisted of a control and 2 challenging formats.

Control: Looking for the best [Service] in [City Name]? Challenger A: Get Free Quotes Today From [Service] In [City Name] Challenger B: [Service] in [City Name] – Get Free Quotes Today

To control for the uncontrollables in search, Thumbtack used the ratio of visits to its experimental titles compared to visits to pages in its control group. Rankings were also monitored, and there was no significant change during the test period. The test ran for 7 days, resulting in a 20-30% drop in traffic to its test groups. Traffic rebounded (for the most part) after reverting to the original title tags.

What to test in ecommerce title tags

If you’re bold enough to try a similar experiment (and have the technical judo chops), here are a few hypotheses you could explore:

• Keyword vs. store name at the beginning of the title
• Short vs. long, “keyword stuffed” titles
• Value proposition such as “Free shipping” or “largest selection of” at the beginning of the title
• "Shop" at the beginning of the title to differentiate from informational results such as Wikipedia articles and blog posts

Remember, to make it work:

• You need a fairly large sample size of URLs that you can experiment, and a tool to support it the test
• Your treatment versions must be tested against your control concurrently, not sequentially
• Allow enough clicks to accrue to reach a statistically significant result
• Understand what you are measuring - are search positions affected (for example, if your brand name at the beginning of the title affects your keyword relevance across the board)? Is traffic affected (can you estimate you are getting higher click through)? Are conversions affected (are you attracting the same quality of visitor)?
• And finally, revert title tags back to normal after the test is over to validate your results

Looking for help with ecommerce strategy? Contact the Elastic Path Research & Strategy team at consulting@elasticpath.com to learn how our ecommerce strategy services can improve your business results.