How to Submit a Complete Sitemap

Google has a new program that enables you to submit the URLs for your entire website, manage the status of your pages, and receive reports about the visibility of your pages on Google. It's called the Google Sitemaps program, and it's worth considering if you manage a large website.

Google Sitemaps serves two general purposes. First, it helps to keep Google informed of all the new and updated pages on your sitein other words, to improve the freshness of the Google index. Second, the program should help to increase the coverage of all your web pages in the Google index. The first goal benefits Google; the second benefits you; and both goals should benefit Google's search users. Participation in Google Sitemaps is free.

Note that the Google Sitemaps program supplements, rather than replaces, the usual methods of adding pages to the Google index. If you don't participate in Google Sitemaps, your pages may still be discovered by the GoogleBot crawler, and you may still manually submit your site for inclusion in the Google index.

Note

Google also has a separate Mobile Sitemaps program to add pages to its Mobile Web Index. Learn more information at www.google.com/webmasters/sitemaps/docs/en/mobile.html.

Note

To participate in the Google Sitemaps program, you first need to sign into your Google account.

Basic Site Submittal

There are two ways you can submit your site for inclusion in the Google Sitemaps program. We'll discuss the easy way first.

If you don't want to bother with submitting a detailed sitemap of your website, you can simply provide Google with the URL of your site. You do this from the My Sites page (www.google.com/webmasters/sitemaps/).

Once you submit your site, you have to verify that you're the site owner. You can do this either by adding a specified <META> tag to the HTML of your site's index page, or by creating and uploading a new HTML page as directed by Google. You do this verification from your new My Sites page (shown in Figure 36.2); click the Verify link and follow the instructions on the following page.

When you submit your site in this basic fashion, Google lets you view a variety of statistics about your site in the Google index. It doesn't, however, guarantee that all the pages in your site will appear in the Google index. For that, you need to submit a sitemap of your site; read on to learn more.

Submitting a Complete Sitemap

To take full advantage of the Google Sitemaps program, you need to submit a complete sitemap of your website. A sitemap is simply a listing of all the pages in your site, URL by URL, with basic information about the contents of each page.

Submitting a sitemap is the preferred way to submit your site, as Google's crawler will never know about all the pages on your siteespecially if some of those pages are generated dynamically or changed frequently. By explicitly telling Google about all the pages on your site, in a format that Google understands, you enhance the chances of those pages being included in Google search results.

Google can accept your sitemap file in a variety of formats (including a plain text file with one URL per line), but the best way to do it is to create an XML file. The file should include the URLs for all the pages in your site, along with optional information about when the page was last changed and the frequency of changes. You should automatically notify Google when your sitemap changes, so Google can work with the latest version.

Note

Even when you submit a complete sitemap, Google doesn't guarantee that it will crawl or index all the URLs on your website. However, since Google uses the data in your sitemap to learn more about your site's structure, it should improve the crawler schedule for your site, and ultimately improve the inclusion of your site's page in Google's search results.

Creating and Submitting Your Sitemap

To create your sitemap, you can use the Google Sitemap Generator tool or avail yourself of the increasing number of third-party sitemap generator tools. I'll discuss these third-party tools in a moment; to download the Google Sitemap Generator, follow the instructions at www.google.com/webmasters/sitemaps/docs/en/sitemap-generator.html.

Once you've created your sitemap, you upload it to your Google Sitemaps account by following these steps:

Using Sitemap Generator Tools

The Google Sitemap Generator is, admittedly, a bit wonky. For most users, an easier way to generate a sitemap is to use one of the many third-party sitemap generator tools, which are quite a bit more user friendly than the tool that Google provides.

For most of these tools, generating a sitemap is as simple as pressing a button. Once the sitemap file is generated, you can then upload it to the root directory of your website and submit it to Google Sitemaps.

Tip

Learn more about Google Sitemaps at the Inside Google Sitemaps blog (sitemaps.blogspot.com).

Some of these tools are web-based, some are software programs, and most are free. The most popular of these tools include the following:

Viewing Sitemap Diagnostics and Statistics

Google Sitemaps provides quite a bit of statistical analysis and some useful diagnostics for your site. You get to this data by clicking the Manage Site link on your My Sites page; this displays the Summary diagnostic page, shown in Figure 36.4.

There are three tabs on this page, with additional pages for each tab. Table 36.1 details what you'll find here.

Table 36.1. Google Sitemaps Diagnostics and Statistics
Tab	Contents
Diagnostics	Summary (indexing summary and problems)
	Crawl errors
	HTTP errors
	Unreachable URLs
	URLs restricted by robots.txt
	URLs not followed
	URLs timed out
	Tools
	robots.txt analysis
Statistics	Query stats (top search queries and search query clicks)
	Crawl stats (page status and PageRank data)
	Page analysis (content type, encodings, and common words on your site and in external links to your site)
	Index stats (links to standard Google site linkage and cache info)
Sitemaps	Management of sitemaps for one or more sites

For example, the Query Stats page, shown in Figure 36.5, lists the top search queries that visitors used to find your site, as well as which of those search queries generated the most clicks. The Crawl Stats page, shown in Figure 36.6, lists the status of pages on your site (successfully crawled, URLs timed out, and so forth), the average PageRank of your pages, and the one page on your site with the highest page rank.