Google explains crawl budget
In 2017, Google’s Gary Illyes explained what crawl budget is, how crawl rate limits work, what crawl demand is and what factors impact a site’s crawl budget.
As Illyes explained, most sites don’t need to worry about crawl budget.
“Prioritizing what to crawl, when, and how much resource the server hosting the site can allocate to crawling is more important for bigger sites, or those that auto-generate pages based on URL parameters,” Illyes said.
More from Illyes:
- Crawl rate limit is designed to help Google not crawl your pages too much and too fast where it hurts your server.
- Crawl demand is how much Google wants to crawl your pages. This is based on how popular your pages are and how stale the content is in the Google index.
- Crawl budget is “taking crawl rate and crawl demand together.” Google defines crawl budget as “the number of URLs Googlebot can and wants to crawl.”
Read all about it in Google explains what “crawl budget” means for webmasters.
Also on this day
Google Ads bug again for Gmail on desktop Safari browsers
2022: This impacted Google Ads that were served in Gmail for users accessing their email using the Safari desktop browser.
Google January 2020 core update almost done rolling out
2020: Google confirmed: “the update is mostly done…”
Google Search Console unparsable structured data report data issue
2020: Some users may have seen a spike in unparsable structured data errors due to an internal Google misconfiguration.
Google Search Console’s URL inspection tool adds HTTP response, page resources, JS logs and rendered screenshot
2019: The tool added the ability to see a specific URL’s HTTP response code, the page resources, the JavaScript logs and a rendered screenshot.
Bing Ads launches page feeds for easier Dynamic Search Ads management
2019: Page feeds eliminated the need to create targets for specific URLs or groups of URLs in DSA campaigns, making it easier to set up and manage.
Google ordered to submit search index to state sponsorship in Russia
2019: A law passed in 2018 mandated that search engine results be filtered through the federal state information system (FGIS).
Katy Jurado Google doodle marks 94th birthday of film star from the Golden Age of Mexican cinema
2018: Jurado was drawn in a powerful pose against a backdrop inspired by the film “High Noon.”
Google Image Search testing YouTube videos
2017: Google was embedding playable YouTube videos for certain queries.
Martin Luther King Jr. Day Google doodle promotes King’s message of unity
2017: The Doodle, by guest artist Keith Mallett, captured one of the major themes of King’s speeches and writing: “All life is interrelated.”
Google Still Doing At Least 1 Trillion Searches Per Year
2015: Google was sticking with the figure it gave in 2012 but stressed it’s “over” that amount. How much? Google wouldn’t say.
Google’s Knowledge Graph May Show For 25% Of All Search Queries
2015: The figure was important because it meant searchers often didn’t need to leave Google to visit a publisher’s site.
2015 Oscar Nominations Now Showing In Google Knowledge Graph & Bing Carousel
2015: Searches for [Oscar nominee] and [Oscar nominations] on Google and Bing delivered results for the best picture candidates.
Local SEOs Sound Off On Google+ Local Hijackings
2014: Local SEOs and people familiar with Google+ Local theorized about on how this happened and shared suggestions o how to guard against future hijackings.
Bing: It’s A Myth That Keyword Rich Domain Names Improve Search Rankings
2014: The domain and keywords in that “domain send less and less” ranking signals to the overall Bing ranking algorithm.
RKG Report: PLAs Fuel AdWords Growth, Bing Ads Spend Rises Sharply In Q4
2014: Paid search spend among RKG’s retail-heavy client set rose 23 percent year-over-year. Overall click volume increased 19 percent, and CPCs ticked up just 3 percent.
Covario: Global PPC Click-Through Rates Hit New Highs, Spend Up 17 Percent YoY In Q4
2014: Based on results from their high tech, consumer electronics and retail clients, PPC spending increased 17% year-over-year and 13% over Q3 2013.
Google Now Cards Hit The Desktop For Mac, Windows and Chrome OS Users
2014: The Google Now cards ran through the latest Google Chrome build (Chrome Canary). Users had to be logged into Chrome.
Dian Fossey Commemorated With Google Logo To Mark The Famous Zoologist’s 82nd Birthday
2014: The logo included illustrations of the Silverback gorillas Fossey studied during her time in Rwanda, along with a picture of the Rwanda mountains where Fossey set camp for most of her 18 years with the gorillas.
Report: PLAs Drove 28% Of Google Non-Brand Ad Clicks In Q4
2013: The non-brand ROI for Google was 22% higher than that for Bing.
Play The Frank Zamboni Google Doodle
2013: The interactive Doodle allows searchers to play the Zamboni game by cleaning the ice after it is being used by children, hockey players and ice dancers.
How To Blackout Your Site (For SOPA/PIPA) Without Hurting SEO
2012: Google’s advice was to use a 503 HTTP status code to tell spiders that the website was temporarily unavilable.
Google Honors Martin Luther King Jr. Day With “I Have A Dream” Logo
2012: The Doodle included a couple lines from his famous speech.
Google Plans To Buy Its Way Into Real Estate
2010: Actually … Google’s goal was to acquire one to two companies a month. Not necessarily just small real estate companies.
Why Don’t Google & Yahoo Offer Twitter Search?
2009: if you want to find Twitter’s powerful, compelling material, you went to Twitter. Not to Yahoo. Not to Google, the king of search.
Microsoft’s Early Mistakes & Missed Opportunities With Paid Search
2009: From bCentral Keywords, to not purchasing Overture, and its failed attempts to buy Yahoo.
Google Names Marchex First ‘Google AdWords Authorized Reseller Technology Platform’
2009: The purpose of this designation was to allow Google to recommend an approved technology partner to AdWords resellers if those partners didn’t have the requisite scalable technology.
Kogan ‘Indefinitely’ Delays Android Phone Launch
2009: Due to technology compatibility issues.
Search In Pictures: Googlers Dancing, Yahoo’s New CEO & Dinner Impossible At Yahoo
2009: The latest images culled from the web, showing what people eat at the search engine companies, how they play, who they meet, where they speak, what toys they have, and more.
December 2008: Search Engine Land’s Most Popular Stories
2009: Search Engine Land’s 10 most popular stories from December 2008.
Google Writes Google Sitemaps FAQs
2008: Google responded to some of the most asked questions on the Sitemaps protocol and how Google supported it.
Google “Online Marketing Challenge” Has 724 College Teams Signed Up To Be Local SEMs
2008: When Google tried to turn college and business school students into search marketers.
Microsoft And EDGAR Online In Content, Ads Deal
2008: Microsoft became the exclusive third-party provider of display, contextual and video advertising for EDGAR Online and its global audience of more than 2 million unique visitors per month.
Yahoo’s Senior VP Of Entertainment Business Leaves
2008: Media executives at Yahoo were leaving one after another.
Hungry For Yahoo? Feed Yourself From Ya-Hoo! Baking Co.
2008: “not affiliated with Yahoo!, Inc. the search engine company.”
Google No Longer Linking To Yahoo, MapQuest Maps
2007: Previously, a search for an address would have yielded a choice among Google Maps, Yahoo and MapQuest.
comScore: Google Wins Again & IE7 Doesn’t Stop Microsoft’s Slide
2007: When Google had just a 47.3% search market share.
New Google Checkout Promo; New Google Trust Worries
2007: Google added the ability to filter product search results from merchants using Google Checkout – and promoted the new feature within its main search results.
Disclosing When Queries Are Autocorrected
2007: It seemed like Google had moved beyond suggesting spelling corrections to automatically doing them for you.
MetaGlossary – Two Million Terms, Phrases And Acronyms
2007: It worked in a similar manner to the Google define: function, though it tended to be more comprehensive.
Wikiseek: Leveraging Wikipedia For Web Search, Poorly
2007: A look at the disappointing service along with a revisit to how it was different from Search Wikia.
Vizu Seeking To ‘Democratize’ Market Research
2007: It offered an interesting model, creating financial incentives for publishers to show polls on their sites and a relatively inexpensive way for marketers to get quick user data.
comScore: Email Top Reason For Mobile Internet Access
2007: “33% strongly agree they ‘feel lost’ without their cell phone.”
Past contributions from Search Engine Land’s Subject Matter Experts (SMEs)
These columns are a snapshot in time and have not been updated since publishing, unless noted. Opinions expressed in these articles are those of the author and not necessarily Search Engine Land.
- 2020: Case study: The true value of informational content for e-commerce SEO by Eoghan Henn
- 2019: Demystifying Google’s guide to clicks, impressions and position in Google Search Console by Glenn Gabe
- 2018: 3 ways to kick-start your SEO in 2018 by Casie Gillette
- 2017: 3 reasons why you can safely ignore your competitors’ traffic metrics by Dianna Huff
- 2015: The Franchise Challenge: How Do You Stand Out When You’re One Of Many? by Rachel Lindteigen
- 2014: At The Digital Crossroads: Shifting To Audience-Centric Marketing by Matt Ackley
- 2014: There’s No SEO Without Mobile SEO by Bryson Meunier
- 2013: How Facebook Exchange (FBX) Can Help Search Marketers Improve Reach & Performance by Frost Prioleau
- 2013: Winning The Conversion But Losing The War by David Rodnitzky
- 2013: How To Get Started With Keyword Research & Content Creation by George Aspland
- 2012: The Enterprise SEO Guide To Response Codes by Ian Lurie
- 2012: 10 Search & Social Resolutions For A Very Mobile 2012 by Brian Klais
- 2012: Paid Search: The Bright-Line Divide by George Michie
- 2012: Simplicity Is Key To Converting Local Consumers To Customers by Myles Anderson
- 2010: Report: It’s A Good Time To Be A Web Analyst, Not Quite So A Google Analytics Competitor by Ben Gott
- 2009: Social Media Gold: Ratings & Reviews by Lance Loveday
- 2009: Q&A With Andrew Silverman, Product Manager For AdWords Conversion Optimizer by Josh Dreller
- 2008: Why You Should Reveal SEO Secrets To Clients by Paul Bruemmer
- 2008 B2B SEO: Capturing Geo-Specific Search by Galen DeYoung
- 2008: When Should You Outsource Your Pay-Per-Click Campaigns? by Jennifer Slegg
< January 15 | Search Marketing History | January 17 >
