Google Sitemaps Review - Google's Performance and Details

Google's Performance and Details

This much of Google Sitemaps is fairly introductory, and I've mostly just shown what you can get from it. However, after using the service for Developer Shed sites like SEO Chat, our web developers had a few questions. Thankfully, after Jill Lindenbaum and Shaluinn Fullove from Google showed me around the service, they helped to answer those concerns.

I first asked about a new site that started using Google Sitemaps and one backlink; after two weeks, the index page was crawled once and the spider did not go deeper. So how should we expect to see Google crawling new sites that use this program and have next to no back-links?

Google representatives explained that new sites shouldn't expect the robot to crawl on demand. There's no way to force the spider to your site, but Sitemaps makes the crawler more effective when it does come. It understands the structure of your site better, and the other pages of the site may be waiting in queue to be crawled. Sure enough, a week later, Googlebot returned and tried to browse pages that were on the site originally and had since been removed; whether from the initial crawl or from the original sitemap, Googlebot was trying to visit the old links it had stored in its database.

We also noticed that Google downloaded the sitemap of our new site at least once a day, but it was not crawling pages. Despite the "lastmod" date and the importance you set for your pages, Google only takes these factors into account as a part of their algorithm. Low PageRank and back-links will tell Google not to crawl often, even if you update the site a few times a day and report this to Google. Googlebot's daily download of our new sitemap may have just been its way of watching the site to learn patterns.

Also, for very large sites that grow by thousands of URLs a day with respectable PR, even reporting all links to Google will not result in having them all crawled. One of our sites (Dev Archives) grows quickly, yet Google's spider is far behind on indexing the pages. Again, Sitemaps is not an on-demand crawler, but the Google reps explained there was more to it than this. Googlebot tries not to overwhelm your server by requesting thousands of URLs a day. It tries to respect your bandwidth, and it should learn the patterns of your site over time. It hasn't learned how to spider Dev Archives yet, but Sitemaps may help it given more time.

The Google Sitemap is considered in the search algorithm, but it does not work on demand. The best way to think of Sitemaps service is as a complement to the crawl, where you can communicate with Google on how to find your information most easily when it does arrive.

Webmasters being able to communicate with Google could promise great improvements to optimization efforts and search results in the future. Still, there are a few things we definitely would like in the future. Google could show more information about the query stats, such as the pages of your site that visitors most often find in their searches and the clickthrough ratio of your "top search query clicks." Showing the pages that link to erroneous locations on your website might also speed up troubleshooting.

That's not to say these features and others are not coming soon. Jill Lindenbaum has alerted me that Google Sitemaps is due for some new features in a few weeks. We don't have information on what these features are yet, but we will be sure to show them off when we have more details.


Search For More Articles!