If Google’s not got all the pages from your site in its index, sorting that out is going to bring you a lot more marketing benefits than anything else you’ll do this week.
Most businesses get most of their website visitors from Google, yet few regularly monitor if their site is indexed properly there. The simple process for checking is to go to Google, get it to list how many pages from your site it has in its index, and judge whether or not that’s about right. If it’s not right, you really need to get on the case. Sorting that out is going to bring you a lot more marketing benefits than anything else you’ll do this week.
So, to start with, you’ll need to estimate how many pages there are on your website. You may just know, or your content management system might tell you. If not, try using Xenu’s Link Sleuth to crawl the site and list them, or if you’re one of our clients, just email us and we’ll do a thorough “site crawl” for you, without charge.
Then go to Google and use the “site:” command, like this (substitute your own domain, with and without the “www”):
You’ll see how many pages Google has got in its index, although bear in mind this is a rough figure. (Note: both queries should show the same number of results, or one should show zero results. If the two queries are showing quite different numbers, you need to contact whoever manages your website and get them to sort out the “canonicalisation problem” using a “permanent redirect”).
If the number of pages in Google is about right, breathe a sigh of relief.
If it’s very wrong, you’ll need to investigate why. I’d throw this over to your website designer, manager or hosting company, rather than agonising over it, as the reasons why can get quite complex. If Google’s number is way too high, the search engine is seeing the same pages with different page addresses, and your position in the Google results will be suffering. If it’s too low, then the site probably isn’t being “crawled” properly, and that needs fixing. Take a look in Google Webmaster Tools (I hope everyone has this installed) as a first step. There may be some reasons quoted there, or at least hints dropped. Then your website manager will need to look at robots.txt, htaccess, noindex meta tags, broken XML sitemaps and connectivity issues: all quite technical stuff which you’ve probably got better things to worry about personally. But do get this fixed.