Whether it is a website rebranding, a consolidation of different web properties or an HTTP to HTTPs migration, when you are implementing a structural web change, it is critical to monitor the crawling, indexing, rankings, traffic and organic search conversions on both the old and new web locations. Careful tracking will enable you to fix any potential problem as they arise.
Besides establishing a relevant strategy to follow that include search engine optimization (SEO) best practices, here are the most important areas and steps to monitor during the web migration stages. Be ready to identify any issues that could cause a negative impact, while also identifying opportunities.
Getting started
Start tracking your organic search visibility on the old and new web locations at least a couple of months before the migration takes place. This will make it easier to identify any unexpected and inconsistent behavior when the change happens.
Old vs. new web crawling
Let’s start with the most fundamental aspects to validate and monitor after executing a migration:
- Can the URLs be crawled in the new web location? Make sure the pages and resources are now effectively accessible in their new addresses.
- Have the 301 redirects from the old to the new URL locations been effectively implemented and tested?
This can be done by using an SEO crawler like Screaming Frog, Sitebulb or Deepcrawl, as well as SEMrush, Ryte or Sistrix, to simulate the search crawlers’ behavior. You can also check (new and old) websites log files.
You can start by simulating both the desktop and smartphone search engine user agents to validate the site’s universal resource locator (URL) accessibility in the new destination, as well as using the “list crawl” option to verify the hypertext transfer protocol (HTTP) status of all the old URLs.
You are looking to see if all the links are 301-redirecting to relevant and correctly configured new URLs. If the links are redirecting correctly, the new URLs will be indexed rather than the old, and the old will not show a 200 OK HTTP or even 404 HTTP error status in the old location, avoiding redirects loops or chains redirects that are more difficult to follow.
To avoid missing old URLs, it is ideal to create a list of them by using different data sources from former pages to migrate and monitor:
- Crawlable URLs in the old location to be found with crawling simulations.
- URLs with any traffic in the last months or year.
- URLs with any top 20 rankings.
- URLs with any external link.
Most SEO crawlers will integrate with other tools. For example, with Deepcrawl you can integrate data from Google Search Console, Google Analytics, Majestic and log files that identify potential crawling gaps. This includes orphan URLs that might not be used anymore but brought value and should be migrated to a new URL version.
It is also advisable to schedule continuous crawls, especially the days before and after the migration happens, and compare with previous results to see if the change is consistent and the redirects haven’t been changed since being set.
I recommended you use a web monitoring service that will send an alert if there are changes in the HTTP status or other SEO-related configurations on the pages you are monitoring.
LittleWarden
Old vs. new web indexing
Monitor the relevant new pages and resources to see if they have been indexed in their new location while the old ones are disappearing from Google’s index. This is easier to do now with the new Google Search Console, where you can add the old and new web locations involved in the migration as independent properties to track.
In the new Index Coverage report, you can see what pages have been effectively indexed from the new location and which have been left out. The report will provide a reason (using a noindex tag, canonicalizing toward others and so on) so you can rectify the problem and re-submit them to the index when the issue is fixed.
The same should be done with the old web location. In this case, check to be sure all of the URLs are 301-redirecting to their new locations and the redirects are going to their new relevant locations.
Old vs. new web rankings
It is also critical to monitor rankings and traffic in the new and old web locations for both desktop and mobile search results. Use both internal and external ranking sources to make sure any rankings lost from the old web location are effectively gained on the new one.
You can easily use the Google Search Console performance report to track countries that are targeted, plus images, videos and rich results.
Monitor the old site for a drop in trend and rankings while the new one is moving up. It is also important to check the performance (with regard to average position and CTR) over time. Although it might take a while to completely gain the old rankings with the new location, you can ensure the new site pages are ranking for the correct keywords and result type (image, video, AMP and so on), especially if you have consolidated the content from one or more pages onto a single page. You’re basically looking to ensure that rankings show a positive trend over time, making certain that there are no gaps in mobile vs. desktop performance or ranked pages.
It is also important to establish a group of your own relevant queries to track and use other rank trackers — such as SEOmonitor, Advanced Web Rankings, SERPwoo and Accuranker — for both the old and new web locations. Your tool should provide day-to-day data updates since Google Search Console has usually a two- or three-day delay in performance data.
In addition, you can track queries which might not be included in the Google Search Console if they are not top-ranked terms but you are targeting them in the SEO process.
When using rank trackers, you will need to have flexibility in the different groups you create to monitor rank behavior. From ranking spread, those queries ranking in the top three to 10 positions to website categories and products, you will want to choose any specific query and see its ranking trend over time. Look in both desktop and mobile results and identify potential ranking behavior issues not evolving as expected in a specific group of pages.
For example, when leaving certain pages in the old web location or when consolidating many old pages into just one, the ranking trend might not be so straightforward. You want to be able to identify this behavior in order to do the proper changes, such as improving the new page’s relevance toward those queries or eliminating/migrating pages you did not move initially.
Ideally, you will also want to track the ranking visibility for relevant keywords against your competitors, to give a context to its organic search visibility trend in the industry before and after the migration.
To complement the monitoring of your most relevant and top queries, you might also follow up with the overall organic search visibility of old and new pages by using external ranking indexes such as Sistrix, Searchmetrics or SEMrush. Doing so helps identify an inconsistent ranking behavior, both in the old and new web location, for terms you are not directly tracking.
When selecting which “external” ranking source to use, find which source has provided consistent historical data with your own organic traffic. It is almost impossible to have complete accuracy, so you might want to use one that has shown a consistent trend over time.
Additionally, not all tools will provide the same degree of data refresh, since many update their rankings on a weekly basis. Their flexibility or granularity to track certain types of web properties differs, so, depending on the type of migration you are monitoring, check if they track HTTP vs HTTPS URLs independently, as well as directories and subdomains for both mobile and desktop results.
Old vs. new web traffic
Last but not least, you need to monitor the evolution of your organic search traffic and the conversions coming from the old and new web locations before and after the migration takes place.
You can use a Google Analytics organic report and easily compare the traffic data after the migration with reports from before in case you have done a migration of the same website or are using the same property identification number.
Check for traffic correlations with the organic search rankings, not only in the overall trend but also in the top pages receiving traffic. Look for specific behavior of users such as bounce rate, time on site, conversion rate and revenue coming from them — all should be the same. And as a bonus, they may get better if page design and optimization are improved after the rankings are regained.
You should be able to integrate your organic traffic data with your tracked keyword rankings to see the traffic and conversions impact. A tool such as SEOmonitor can help with this.
Although setting a tracking system to monitor the crawling, indexing, traffic and conversions from an old site to a new one can take time and resources, it will help you understand and quickly identify issues to fix during the migration process.
Contributing authors are invited to create content for Search Engine Land and are chosen for their expertise and contribution to the search community. Our contributors work under the oversight of the editorial staff and contributions are checked for quality and relevance to our readers. The opinions they express are their own.
