I saw a discussion on Twitter this morning about the idea of having a feature in Robots.txt that would block both crawling AND indexing. It started with this tweet by Christian Thurston (@ct_oz):
“Hi John [Mueller], has Google considered making it so that the robots.txt file doesn’t just block crawling, but also blocks indexation? To quote @willcritchlow: “I can’t see many situations where I want to block crawling but don’t want to block indexing”.
“That would be a significant change in expectations (and yes, we do think about these things regardless). Do you have some examples where this would cause a visible improvement in search?” Mueller responded. “I’d like to avoid adding more directives. I’m still not aware of common issues caused by this documented functionality … SEOs worry about indexing, but usually these URLs only rank for site:-queries (or if there isn’t other, better content on the site), so it feels artificial?”
With over 20 years of experience in SEO, “I have never encountered a situation where a publisher wanted to have a page indexed that they block for crawling in robots.txt. Not even once have I seen that,” I tweeted in response. “It’s common practice for me to educate people that they have a choice: (1) block crawling, or: (2) prevent indexation, when what they want to do is both. Note: definitely more of an issue for larger sites where crawl budget is an issue.”
Will Critchlow of Search Pilot agreed, “100% agree. I can see the conservatism of not changing a long-standing standard, but I have never seen, and can’t think of a situation where you’d want to block crawling but allow indexation.”
But what do YOU think? Would it be helpful to have a feature in Robots.txt that allowed you to specify the pages you don’t want to have indexed? Take our quick three-question poll below and let us know what you think.
Please keep in mind that if you vote yes for doing this that you would need to accept ALL the risk that at some point your dev team might misapply the capability and NoIndex pages that you don’t want them to.
Loading…Loading…
Contributing authors are invited to create content for Search Engine Land and are chosen for their expertise and contribution to the search community. Our contributors work under the oversight of the editorial staff and contributions are checked for quality and relevance to our readers. The opinions they express are their own.
