As expected, Google removed the crawl rate setting tool within Google Search Console. Last November, Google told us they would deprecate the crawl rate limiter legacy tool within Google Search Console on January 8, 2024.
January 8th came and went and so did the crawl rate limiter with it.
Google said this feature is no longer useful because Google said it has improved its “crawling logic and other tools available to publishers.”
It’s gone. Now when you try to access the tool for a specific property within Google Search Console, you are redirected to the notice of the tool going offline.
What was the crawl rate limiter. It is a tool within the legacy version of Google Search Console that lets you communicate to Google crawl your site less than it currently does. Google has historically recommended against limiting the crawl rate unless you are seeing server load problems that are definitely caused by Googlebot hitting your server too hard.
This is what the tool looked like:
Why Google removed the tool. Gary Illyes from Google said, “with the improvements we’ve made to our crawling logic and other tools available to publishers, its usefulness has dissipated.”
“Googlebot reacts to how the site–or more specifically the server handling the site– responds to Googlebot’s HTTP requests. For example, if the server persistently returns HTTP 500 status codes for a range of URLs, Googlebot will automatically, and almost immediately slow down crawling. Similarly, Googlebot slows down automatically if the response time for requests gets significantly longer. If you do experience unusually heavy crawling that your site can’t manage on its own, refer to this help article,” Gary Illyes added.
Illyes explained that the rate limiter tool within Google Search Console “had a much slower effect.” He said the tool often would “have taken over a day for the new limits to be applied on crawling.”
He added that the tool was “rarely” used by site owners and those who used it “in many cases set the crawling speed to the bare minimum.”
Crawl rate change. Google said with the deprecation of the crawl limiter tool, Google is setting the minimum crawling speed to a lower rate, comparable to the old crawl rate limits. So Google will “effectively continue honoring the settings that some site owners have set in the past if the Search interest is low, and our crawlers don’t waste the site’s bandwidth,” Illyes added.
So hopefully you won’t need to use this tool anyway.
What if you have issues. If you have issues with crawling, Google said you can read this help document and use the report form to let Google know.
Why we care. If you have been using this crawl rate tool, keep in mind it no longer works and there is no way to access the tool anymore. You may want to check how this change impacts your crawl rate going forward, over the coming days and weeks.
