20 C
New York
Thursday, May 23, 2024

Search Engines Unite On Sitemaps Autodiscovery

Last November,


united to support sitemaps, a
standardized method of submitting web pages through feeds to the search engines.
Today, the three are now joined by Ask.com in supporting the system and an
extension of it called autodiscovery. This is where the major search engines
will automatically locate your sitemaps file if the location is listed in a
robots.txt file. Announcements are up from

and Ask
now Yahoo and

Information on how to create sitemaps files can be found at the
Sitemaps.org site. Aside from the
sitemaps XML formal, you can also provide RSS 2.0 or Atom 0.3 or 1.0 feeds.
That’s handy for those with blogs that already generate these feeds.

Sitemaps XML files too complicated? Don’t run a blog? Note that the site has
newly expanded information on how you can submit a simple list of URLs in a text

In the past, if you created a sitemaps file, you then had to manually tell
the search engines where to find it. With today’s announcement, search engines
will check your
robots.txt file
for a link to a sitemaps file, then get the file from that
location. This is a big plus because all the major search engines regularly
check robots.txt files as part of their ordinary crawling.

To add the location, just put a line like this anywhere in your robots.txt


Replace the LOCATION-OF-SITEMAPS-FILE with the actual location. For example,
if you ran a site at mywonderfulsite.com and had a sitemaps file called
allmypages.xml in your top level, the reference would be like this:

Sitemap: http://mywonderfulsite.com/allmypages.xml

Have more than one sitemaps file? Ideally, you’d create a special "sitemaps
index" file that links to all of them, then put a link to the sitemaps index
file in your robots.txt file. If that sounds like too much work, you can have
more than one sitemaps URL listed in the robots.txt file.

Aside from autodiscovery, you can also ping Google and Yahoo with the
location of your file. The Sitemaps.org site has more instructions on this in
general. For specifics:

  • Google: See

    . Note that this pinging is different than the pinging Google
    for blog search.
  • Yahoo: See

    . Unlike Google, the same pinging system is used for both web and blog
    search, to my understanding.

Both Google and Yahoo also allow you to manually submit sitemaps files. In
both cases, doing this via their
Google Webmaster Central
Yahoo Site Explorer
systems gives you access to specialized monitoring and
reporting tools or information on how they crawl you.

For more about these tools, or how each individual search engine handles
sitemaps files, please see the links below:

Keep in mind that Microsoft and Ask are still lacking references to sitemaps
information, but I expect this will change over time.

For related coverage, see here and here on Techmeme.

Related Articles

Latest Articles