Google recently released news that it is now moving to encrypt all organic search activity. This means that analytics programs will no longer be able to report on keyword terms that drive organic traffic from Google, instead noting the data as being “not provided.”
This could be devastating to marketing managers and webmasters who rely heavily on this search data to determine ROI on their organic campaigns. With the data obfuscated from analytics programs, many feel that the organic search marketing channel has been stricken from the record as a legitimate marketing channel.
This data obfuscation should serve as a clear indicator to marketing managers that they must begin using organic search to fuel a healthy, more holistic inbound marketing strategy. We took the indicators that way. But what if you could still get access to the “not provided” data?
Having an idea of keyword-specific organic traffic to landing pages is more invaluable than ever. Understanding this data helps a marketing manager correctly focus organic budget and efforts. So how would one go about calculating keyword-specific data if 100 percent of your traffic will show up as “not provided” moving forward?
Uncloaking Not Provided
Use Case: Reporting on last month’s keyword-specific organic traffic to a landing page.
Step 1: Pull traffic data for a landing page from Google Analytics for the previous month.
To do this, navigate to the “Behavior” reports, drilling down into “Site Content” and viewing the “Landing Pages” report. From here, set your date range to the previous month. You can see all landing pages that received traffic over the last month as well as how much traffic they received. Export the data, and step 1 is complete.
Step 2: Build an irrefutable list of keywords that drive traffic to a targeted landing page. Identify — and, moving forward, periodically catalog — keywords and keyword rankings for specific landing pages using a tool like SEMRush.
You should get a fairly accurate snapshot of keywords driving traffic to specific landing pages. SEMRush also provides rankings. SEMRush updates their data once a month; if you’d like more frequently updated data for a more detailed look, you could use another tool (easy to find) that pulls rankings more regularly like once a week or even once a day. Export the data, and step 2 is complete.
If we were doing this with existing pages, we’d start with analytics to build the list, supplement the list with paid search data, and compliment it with Bing/Yahoo search volume data using methods similar to Moz’s explanation. If we were doing this with new pages, we’d start with SEMRush data and supplement it with data from a host of other keyword ranking identification tools.
Step 3: Pull search volumes for the keywords ranking for the landing page from Google Keyword Planner (yes, you can pull exact and phrase).
Google Keyword Planner recently replaced Google Keyword Tool. Yes, the data is still available. Here’s how to find and use it: Go to Google Keyword Planner and click the “Get traffic estimates for a list of keywords” option. Drop in your list of keywords, which were pulled from SEMRush. Turn bid range all the way up. Click the last point on the bid range.
The table you’ve produced will show a list of your keywords with daily estimate data. For exact match search volumes, go to the “Edit match types” dropdown, hover over “All ad groups” and select “Exact match.” Make sure you select “Segment by” keyword. Download the data.
For phrase match search volumes, go to the “Edit match types” dropdown, hover over “All ad groups” and select “Phrase match.” Make sure “Segment by” keyword is selected. Download the data. To calculate monthly data, take the daily estimates for both your phrase and exact match, multiply each keyword data point by 365 (for days in a year) and divide by 12 (for months in a year). Then, to get exact percent of search volume, divide your exact monthly impressions by total of exact and phrase monthly impressions.
Do the same for phrase percent of search volume, divide your phrase monthly impressions by total of exact and phrase monthly impressions. Multiply each percentage by the search volume provided by the Google Estimator. This will give you monthly search volume for both phrase and exact for that keyword.
Step 4: Determine CTR ratios, using a CTR study like the ones from BrightEdge or DigitalRelevance as a benchmark.
Take the keywords and keyword rankings that you pulled from step 2, apply CTR to the rankings to get a CTR benchmark for exact and CTR benchmark for phrase. Multiply the monthly exact search volume you pulled from step 3 to the exact CTR benchmark for each keyword. Multiply the monthly phrase search volume you pulled from step 3 to the phrase CTR benchmark for each keyword. This will give you a traffic estimate benchmark for each keyword.
Let’s talk for a moment about one of the major benefits of using a CTR study that includes LT C
TR. Long-tail CTR is extremely important when understanding total traffic to a landing page. Long-tail traffic generally makes up the majority of traffic to a landing page. In other words, 5 percent of phrase volume from a rank-1, page-1 search engine listing will be much more traffic than 18 percent of exact volume to the same listing. That’s why it’s essential to have long-tail CTRs included in calculations when uncloaking not provided data (and similarly, when using predictive analytics to provide traffic estimates).
But what about keywords that don’t fall into positions 1-10 because they’re covered by the CTR study? Here’s where regression analysis comes in. Take the CTR study curve, apply a regression trend line to the curve, and you have your CTRs for the positions that follow. From here, perform the same calculation mentioned above for your keywords that are in positions higher than 10.
Step 5: Perform calculation (yes, this is provided in the post) to determine keyword-specific organic search traffic.
Total all the traffic benchmarks from Step 4. Divide the traffic for each keyword by overall traffic to get a percent of overall traffic for that keyword. Compare your total traffic to the Google Analytics landing page traffic pulled in Step 1. There will most likely be a difference — and that’s fine. The CTR study is a benchmark. Your site may have under-performed or out-performed the benchmark.
To determine each keyword’s actual traffic from here, multiply the total landing page traffic by each keyword’s percent of overall traffic. Voila! You have successfully uncloaked keyword-specific organic “not provided” data with a degree of accuracy based on an industry standard.
Formulas
Below are the formulas for uncloaking not provided data versus provided data and data results.
Uncloaking Not Provided Data
- Exact CTR: Use Current Rank to Pull Exact CTR from CTR results tool
- Exact Monthly Search Impressions: (Exact Daily Impressions * 365)/12
- Exact % of Search Volume: Exact Monthly Search Impressions/(Exact Monthly Search Impressions + Phrase Monthly Search Impressions)
- Search Volume Exact: Exact % of Search Volume * Google Provided Search Volume
- Not Provided Exact Volume: Search Volume Exact * Exact CTR
- Phrase CTR: Use Current Rank to Pull Long-Tail CTR from CTR tool
- Phrase Monthly Search Impressions: (Phrase Daily Impressions * 365)/12
- Phrase % of Search Volume: Phrase Monthly Search Impressions/(Exact Monthly Search Impressions + Phrase Monthly Search Impressions)
- Search Volume Phrase: Phrase % of Search Volume * Google Provided Search Volume
- Not Provided Phrase Volume: Search Volume Phrase * Phrase CTR
- Not Provided Total Traffic (driven by that keyword): Not Provided Exact Volume + Not Provided Phrase Volume
- Not Provided % of Overall Traffic: Not Provided Total (for that keyword) / Sum of Not Provided Total (sum of all keywords)
Versus Provided Data
- Actual Traffic (for each keyword): Pulled from Analytics
- Not Provided Calculation (for each keyword): Actual Traffic * (1-Not Provided (for the month)/Sum of All Keywords Actual Traffic)
- Total Traffic Per Keyword (better estimate for each keyword): Actual Traffic + Not Provided Calculation
- % of Overall Traffic: Total Traffic Per Keyword / Sum of Total Traffic Per Keyword (sum of all keywords)
Data Results
- Diffs: Absolute Value of (Not Provided – Actual)
- Traffic for Not Provided Final Breakdown (for each keyword): Overall Landing Page Traffic * Not Provided % of Overall Traffic
Uncloaking Not Provided Next Steps & Considerations
We will work to refine this method for more accurate results as time progresses. For now, this should serve as a basic starting point for calculating not provided moving forward. We’ve worked on putting this uncloaking methodology together. While it still needs some work, our next steps are meant to provide real-world examples.
For more accurate results, build a stronger keyword list for the targeted landing page by using additional keyword research tools. In addition, the same analysis would be better by tracking daily SERP rankings for each keyword using a platform like BrightEdge.
In turn, CTRs would change daily and the overall calculation would provide a more detailed picture of the not provided organic search traffic by keyword breakdown. Regarding each data pull (keyword list, analytics data, keyword planner, etc), the goal is to build a tool to pull this data periodically and perform calculations automatically after providing only a targeted landing page.
Using this theory of uncloaking “not provided,” we’ve run some initial real-world tests and have found a positive correlation. Although there is a positive correlation to the theory compared to the actual data, the data does not conclusively support this method of uncloaking “not provided.” Simply put, we need more tests and more data to provide irrefutable evidence of using this method. More importantly, the goal here is to spur discussion around this topic and hopefully provide a starting point for keyword-specific reporting and decision-making.
Contributing authors are invited to create content for Search Engine Land and are chosen for their expertise and contribution to the search community. Our contributors work under the oversight of the editorial staff and contributions are checked for quality and relevance to our readers. The opinions they express are their own.
