Subscribe via RSS FeedStop Google Analytics From Stealing Your Valuable AdWords Keyword Data
Are you a Google AdWords advertiser using Google Analytics? STOP! You MUST read this post because you are losing money daily and we are going to help you stop the bleeding.
There is a problem with the default functionality of Google Analytics when used in conjunction with AdWords. Google Analytics (GA) doesn’t report the actual phrase a shopper entered into the search bar, only the keyword phrase you are bidding on.
Let me explain:
- You bid on the keyword ’shoes’ using ‘broad match’
- A shopper searches for ‘blue suede shoes’
- The Traffic Sources > Keyword report in GA shows the search as just ’shoes’
Even worse, Google likes to use synonyms when your terms are under the broad match type (called automatic matching or extended broad match).
- You are bidding on the keyword ‘running shoes’ using ‘broad match’
- A shopper searches for ‘Adidas Gazelle’
- Google shows your ad, but wait, you don’t carry Adidas shoes
Why would Google do that?
The shopper searched on blue suede shoes, not shoes! You don’t sell blue suede shoes. You have been making decisions based on inaccurate data.
Follow these simple steps to start seeing the EXACT phrases people are using when they click your AdWords ads.
It will help you find terms to add to your negative keyword list. You can also start honing your ad and landing page copy to better reflect how shoppers search.
The Google Analytics Exact Query Solution…
This solution comes from our friends at VKI Studios, a Google Analytics Authorized Consultant and overall great bunch of people (see their analytics blog for some great tools and tips). Specifically, Brian Katz. They have evaluated various means of cracking this nut, and we have their final solution. Credit and comparison of other methods are at the bottom of this tutorial.
1. Create a new Google Analytics Profile
We do NOT want to overwrite any core data, so a new profile keeps everything intact. Even Google says it is a good idea.
Select Add a Profile for an existing domain, select which domain, and enter any name for the profile you choose - the more descriptive the better. You will not have to add any tracking code or tag anything, so no need to get the ponytail guys involved.
2. Create the first filter
Locate your newly created profile and click Edit under the Settings column. Then click Add Filter.
Field A -> Extract A: Referral: (\?|&)(q|p|query)=([^&]*)
Field B -> Extract B: Campaign Medium: (cpc|ppc)
Output To -> Constructor: Custom Field 1: $A3
3. Create the second filter
Locate your new profile again and click Edit under the Settings column. Then click Add Filter.
Field A -> Extract A: Custom Field 1: (.*)
Field B -> Extract B: Campaign Term: (.*)
Output To -> Constructor: Campaign Term: $B1 ($A1)
As with almost all multi-part filters, sequence is critical and must be ordered accordingly using the “Assign Filter Order” page for the profile.
That’s It!
Here are what the results should look like when you run the Traffic Sources > Keywords > Paid report in Google Analytics:
The following set of results were obtained using an in-line filter to show bid-terms that would be different from the search terms
An unfiltered result would look as follows:
The above technique provides useful data as is but it does have some shortcomings in that it does not associate the newly overwritten Campaign Term field with Transactions, as is shown in the following screen shot:
It is probably the result of using session-based values (e.g.: all the Campaign fields) and pageView-based values (e.g.: Referral). Caught in the middle are the event-based eCommerce transactions.
In his book “Advanced Web Metrics with GA” (Page 199) Brian Clifton documents a method attributed to Shawn Purtell of ROI Revolution that uses 3 filters to show each Transaction with its bid and search terms appended.
We are experimenting with a combination of those filters and the ones described above to extend the solution to include eCommerce and will post the solution when we have it. So make sure you subscribe to the RSS Feed or by Email to be notified when it is available.
Hat Tips to Others Tackling this Problem
The original solution for this came from Brian Clifton, formerly of Google.
The solutions (Using Filters):
- How to Get Detailed PPC Keyword Data from Google Analytics
- NUDE: AdWords Keyword Data Exposed With Google Analytics!
An updated solution from ROI Revolution (Using JavaScript):
This solution uses the User Defined variable so it won’t be appropriate if you’re using the User Defined variable (created with _setVar()) already
- Exact Keyword Tracking with Google Analytics, Revisited
- Exact Keyword Tracking with ga.js
Comparison of the two methods
I checked out the two methods (Filters vs. JavaScript) . Since readers commented saying the filters did not work or “no longer worked”, I took a closer look. The devil is in the detail. Errors in their implementations may have been the cause of the malfunctions.
JavaScript vs. Filters
JavaScript
The two methods both extract data from the and Referrer and Campaign Medium checking the latter for “ppc” of “cpc” using regular expressions. They both concatenate the bid and search terms. The JavaScript method goes 1 step further by looking for the gclid value unique to Google AdWords. That may also be done in the filters but I don’t believe it would enhance the filter solution.
The JavaScript performs its magic at run time. It uses the “troublesome” _setVar() cookie to store the bid and search terms in the User Defined field. It does so using a generally accepted “kludge” to work around _setVar()’s issues (a topic all of its own).
The greatest disadvantage to this method is that it monopolizes the User Defined Value. With all its troubles, it is an invaluable resource that most will (should ?) be using to segment visitors. Since it is stored in a domain specific cookie it cannot store profile-specific data to different profiles (well, it can be pushed to greater limits but that is a blog post all of its own).
It should be possible to rewrite the URL of the landing page before ga.js writes the Campaign cookie (again a topic of its own)
Filters
The filters run at data processing time and so, I expect those may prove marginally more reliable than JavaScript and cookies (although all subsequent visits from the AdWords campaign will rely on the keyword and other campaign data being extracted from cookies by ga.js or urchin.js) so that is no reason to choose one above the other.
By default, however, I am biased in favor of filter-based solutions because they are independent of the implementation and so don’t require updates to a site’s GA coding. Implementation is quicker and easier, as is propagation of the solution across profiles and GA accounts. In fact, in the time it takes to update the code on some sites (those that are not tagged as efficiently as they might have been) or in the time to get a site’s 3rd party developers to make the changes, a GA consultant could implement the solution for a number of accounts, regardless of the level of access the consultant has to the coding.
Note: Analysis and much of the technical write-up done by VKI Studios, Brian Katz
Like This Article?
|
Get New Posts Delivered to You
|
Add to del.icio.us
Sphinn It!
Stumble It
This is amazing post, Linda! I was always upset that it did not really show well the actual data that people are searching. AND, now there is a so much better way to use negative adwords!
@ Janis, yeah it is amazing - but this one’s all Jason B’s and VKI Studios’ work.
:)
Great work, I’ve always wondered why this wasn’t available by default. This really should be added to the 8 Stupid Things Webmasters Do (and don’t know about) that Mess Up Analytics. http://www.marketingpilgrim.com/2008/05/8-stupid-things-webmasters-do-to-mess-up-their-analytics.html
This would be #1 for anyone using broad and phrase matching.
I agree, Linda, it should be added by default. One side to that is that Google would earn less millions if suddenly we would have a better way to minimize wasted expense on PPCs. But give it a year or Yahoo to do that, and Google will catch up eventually.
PS. My bad about the authors! Good job, guys!
Thank you so much for this post. I’m a heavy user of Adwords and didn’t even think to look into this, despite being aware that it was a broad match.
Hat tip for the excellent information. This should save everyone using AdWords some serious money — money that can be well spent in other marketing areas.
Great topic Jason, a lot of people I’m sure had no idea this was going on and were making a lot of il-informed decisions.
You guy’s did a great job breaking down the steps to make it easy for anyone to implement. I typically run Search Query reports to discover what keywords Google is connecting to my Broad Term campaigns but this new GA profile should put some light on the “Other Unique Queries” row of the spreadsheet.
Great Post!
Duane
Great post Jason. I’ve known about this for a while but didn’t know a solution to fix it.
I have to re-iterate. It was a bunch of people who contributed to this but mostly the guys at VKI Studios. http://www.vkistudios.com
Nice assessment and review, thanks for the great analysis. Also the link to the 8 mistakes post, which is one I haven’t seen before.
I gotta jump in and defend Google Analytics here for a second, they give all this information for FREE - perhaps this hack will become a feature soon, sometimes it takes demand before something gets developed. Everyone email your AdWords Client Rep :)
Google being the “open” company that it is (evidenced of increasingly sharing its own information including keyword search counts now), I wouldn’t be surprised if it fixed this problem. The more relevant the results, the more money Google makes. If searchers don’t like search results, they’re less likely to click on them ever in the future - if they have regular bad experiences.
Also, if advertisers look at ROI and search doesn’t perform well, in this economy, they might scale back on search all together. Even Google should fear that.
If you find a lot of weird and irrelevant referrals, you might need to tweak your ad copy too, for some reason you’re not making yourself clear (through price or correct offers/description of your products) enough to deter clicks. Not all Google’s fault!
But this hack helps you find if that’s the case, and adjust your campaign accordingly by negative matching, tweaking AdGroup organization and ad-copy.
now i understand why most broad search terms get high bounce rate… Thanks for sharing this wonderful information…
I’d highly recommend moving the actual search term to the user defined field rather than appending it to the bid search term. This approach allows you to select a given bid term (thermos,dvd) and then cross segment that keyword by user defined to see all the raw search terms for that bid term without having to memorize the regex filter from this article. It also allows you to look at the user defined report to see a complete list of raw search terms. Keep in mind that eventhough this works most of the time, about 25% of the time the actual keyword will show as not set.
If you’re not using this technique to find negative keywords, you really are wasting money.
One thing to be aware of with the filter solution to this problem: using a filter in Google Analytics does not actively set a cookie in the visitor’s browser, so if they happen to come back via another method–one that does not have a q or a p query parameter and a search query–then the visitor will not be tagged as such.
The issue is that the exact query that the visitor used to arrive at your site is available in Google Analytics’ Referral field only on their initial visit to the site. So, if you use a filter to grab that data, it will only last for their first session and until they come back to your site. At that point, the query no longer exists in the Referral field.
For sites where conversion rarely occurred on first-time visits, it seemed necessary to set a cookie that would persist across multiple sessions. This is why I wrote the script version mentioned above.
The purpose behind our contribution to the article was to highlight the differences between the various methods - which were not ours but originated by luminaries like Michael Harrison (ROI) and James Zolman (NUDE - semvironment) and Brian Clifton.
Michael, thanks for making the point I did not - that of persisting the referral.
Brian Katz
VKI Studios - Google Analytics Authorized Consultants
http://www.vkistudios.com
Could anyone tell me if this hack should work instantanious, or if it only works after the filter is installed on newly aquired data?
If the latter is true, I probably did something wrong..
The latter is true, Nick. It should only work for data collected after you apply the filter.
You do realize that between your genius, original post on 8 common analytics mistakes and this, you’re quickly branding yourself as an analytics pro :D? Do you do visual sciences too btw?
@Brian Katz from VKI Studios -
I have to admit that we found it from ga-analytics.co.uk…who found it from Brian Clifton who was originally with Google (as mentioned at the end of the post). We simply rewrote it with a bunch of screen shots and a step by step guide at the beginning of the year on semvironment. I wish we could claim some credit but we really can not…ROI Revolution is awesome for writing/creating the JS fixes…and Brian Clifton definitely deserves all original credit!
Thank you for mentioning my name though - and next to the complimentary word ‘luminaries’…you are way too kind! :) It’s always good to revisit such a post and make it a little bit better. It would be nice to get a video/screencast of this tip…maybe we’ll do one soon. ;)
I almost forgot…thank you for the hat tip towards the end of the post Jason! We’re big fans of Get Elastic/Linda/and you! Keep up the great writing and work. :)
Great information. This is exactly what I was looking for.
No one seems to want to take the blame, sorry CREDIT, for this. :D
I was aching to find a solution for this and had an endless closed loop conversation with Google Support which went nowhere. Had to install Free Stat Counter finally which was such an inelegant solution *blush*
Thank you, Gracias, ありがとう, Danke, Merci, Grazie… very mucho.
Added to my RSS drip feed. I’ll be back.
I’m exploring the referred links and also the links coming in. What a treasure trove, had to create a separate tab on my Google Home page for this.
@Rakesh, welcome, glad you like it and thanks for adding us to your RSS.
I do have a problem. I did what you said, but for the new profile it still says “Tracking Unknown” and no data on the reports. Do I have to add a new code on the pages?
Thanks for help.
Solved. I did a mistake.
Thanks for information.
Thanks for this wonderful info!
I really love these new reports.
Thanks a lot!
“No need to get the ponytail guys involved” — that’s the funniest thing I’ve heard all day. Nice work folks, I’ll be deliciousing (a word? it is now) this post for years to come.
Hi, this is a really great post!
Can anyone help me by any chance? I’ve set everything up as needed, but the tracking code for the new profile cannot be found. I think it is because when i create a new profile the tracking code at the end changes to a different digit (e.g. usual filter: UA-153940-7, newly created filter UA-153940-8. Or maybe i’m missing something :(