Google Webmaster Tools (GWT) is a free resource that allows site owners to get actionable information about their sites’ performance in Google results, including click-through rates from search results pages, site errors that may depress organic traffic, penalties and warnings about sites being compromised, and indexing status.  GWT also includes tools for testing a site’s robots.txt configuration, removing unwanted URLs from Google’s search index, submitting XML sitemaps, and more.

Together with your analytics data, information from Google Webmaster Tools can help you prioritize technical and marketing efforts to improve the visibility (and therefore traffic) your sites get from search engines.

Below are steps for getting started.

I. Setting up your Google Webmaster Tools Account

II. Adding additional users/emails to your GWT Account

III.  Adding XML Sitemaps

IV. Using GWT to remove URLs from Google’s Index

I. Setting up your Google Webmaster Tools Account

 

To create a new account, go to: www.google.com/webmasters/tools/ 

  1. Log in using the Google account you want associated with GWT. This does not need to be a gmail address; it can be your work email address, as long as you’ve registered it as a Google account.
  2. If this is a new GWT account with no sites, fill in the box on the landing page with the domain name you want to track. Use the complete name, including www-subdomain, if relevant. (If you enter station.org, GWT interprets this as http://station.org and will not be able to show you any data if your site actually lives on http://www.station.org/
  3. Follow the instructions for verifying site ownership. If you can’t verify using the “recommended method”, click the tab for “alternate methods” and choose one of the other options.
  4. You can add multiple sites to your GWT account. If you’re using a GWT account that already contains one or more sites, click the “Add a Site” button to a new site. Current instructions for adding a site to Google Webmaster Tools can be found here.

II. Adding additional users/emails to your GWT Account

 

Once you’ve set up a site on Google Webmaster Tools, you can add other users (e.g., co-workers, consultants) to the account on a site-by-site basis. If you opened the account using an email address you don’t check daily, you should consider adding an alternate email that does get checked regularly because you don’t want to miss critical messages from Google about your site.

 

  1. To add new users, start by logging in to Google Webmaster Tools and go to your main dashboard here: https://www.google.com/webmasters/tools/home (Note: only the “owner” account can add other users)
  2. To the right of each site listed in your account, find the dropdown menu called “Manage Site”
  3. Click the “Add or remove users” link in the dropdown
  4. In the upper right, click on the button that says “Add a New User”
  5. Enter the email of the account that should have access. (This needs to be a Google-registered account, but does not have to be a gmail address. I recommend a work email.)
  6. Select whether this user shall have “restricted” or “full” access. Details about permission levels can be found here: https://support.google.com/webmasters/answer/2451999?hl=en
  7. Repeat these steps for every site/host in the account as desired.

 

III.  Adding XML Sitemaps

 

As Google explains, “Sitemaps are a way to tell Google about pages on your site we might not otherwise discover. In its simplest terms, a XML Sitemap—usually called Sitemap, with a capital S—is a list of the pages on your website. Creating and submitting a Sitemap helps make sure that Google knows about all the pages on your site, including URLs that may not be discoverable by Google's normal crawling process.” By submitting a dynamic Sitemap, you can also help speed up Google’s discovery of new pages.  Learn more: https://support.google.com/webmasters/topic/8476?hl=en

If your CMS does not auto-generate an XML Sitemap, you can create your own (acceptable Sitemap formats include .txt, .xml, .rss, .mrss) and upload it to your site. Keep in mind that Sitemaps you’re creating and uploading by hand (in contrast to dynamically generated feed) will not help Google discover new content you add later unless you keep updating those files; in that case, you will need to keep updating those files if you’re adding content Google can’t discover naturally.

To submit a Sitemap:

  1. Locate where your site’s XML Sitemap exists. By default, some content management systems place the Sitemap at the root and call it sitemap.xml. For example: www.yourdomain.org/sitemap.xml
  2. Log into Google Webmaster Tools and go to the site admin for the site you’re working on.
  3. In the left nav, expand the “Crawl” section
  4. Click on “Sitemaps”
  5. In the upper right corner, click on “Add/Test Sitemap”
  6. Enter the URL path of the location where your Sitemap exists
  7. Click “Submit”
  8. Check back later to check the status of the URLs you submitted

Keep checking back to make sure your Sitemap isn’t throwing errors and that Google can access the URLs you’re submitting.

IV. Using GWT to remove URLs from Google’s Index

 

On occasion, pages or images from your site may end up in Google’s search results that you decide shouldn’t be there anymore. Even if you delete or replace that asset (whether it be an image or an entire page), a copy of it may live on in Google's index for a long time.

To remove a URL from Google entirely, here are the steps you need to take to take: 1) remove the asset from the site and 2) submit a URL removal request in Google Webmaster Tools.

                                                            

1. Remove the asset from your site

There are three different steps you can take to meet this requirement:

 

  • Delete the asset and make sure the associated URL returns a 404 or 410 status code
  • Block the content using a robots.txt file
  • Block the content using a meta noindex tag (only works for HTML pages, not images)

Only the first option also ensures that no human beings will see it either. The second two options are requests to robots not to include the files in their index – but not all robots honor those requests – and they don’t prevent any human visitors from seeing the content if they know the URL. Also, anyone can look inside a robots.txt file to see all the directories and files you don’t want to expose, so that’s not a great way to hide something you really don’t want anyone to see.

If you don’t want users or search engines to be able to see or find a particular file on your site, then the best thing is to delete it outright and make sure that the server tells the browser the asset no longer exists.

 

2. Submit a Removal Request to Google

To initiate a removal request:

 

  1. Log into your Google Webmaster Tools Account
  2. From the main dashboard, click on the site associated with the asset you want to remove (This would be the domain that the asset resides on); if your site isn’t already listed as verified in your account, you must first follow the instructions to verify your site.
  3. In the left nav, expand “Google Index”
  4. Click on “Remove URLs”
  5. Click the “Create a new removal request” button
  6. Enter the URL of the asset you want to remove from Google or the path of the entire directory you want removed
  7. The next page will give you three options in a dropdown:
    • Remove page from search results and cache
    • Remove page from cache only
    • Remove directory

Pick the most appropriate option for what you’re trying to accomplish.

Your URL removal request will now show up in the “pending” list until the request has been processed. Check back in a couple hours to see if the request was approved and then go to Google to make sure the page or image is indeed gone. (It’s important to understand that this is an automated process--not a manual review --so be sure to select the appropriate options and follow the instructions).

To request cache removal of content on a domain you do not control:

  1. Verify that the URL of the content which you wish to have removed from Google’s index either returns a server error or is blocked from robots.
  2. Go to https://www.google.com/webmasters/tools/removals?pli=1
  3. Click “create a new removal request”
  4. Enter the full URL of the page you want removed
  5. Click “yes, remove this page” when prompted.
  6. Repeat for any additional URLs.