Presenting PDF documents to the web: a summary

Harder than this is getting a page removed from the index completely. Just deleting a page from your site is a hopeless approach; the search engines might take months of crawling and re-crawling your site before they decide that the page really no longer exists. You need to flag the page as having gone, either by redirecting its URL to a replacement page, or listing it as gone, which in Google, you do through Webmaster Tools.

(You have got a Webmaster Tools account, haven't you? Whoever created your website should have ensured you do, as part of the service, or they weren't doing their job properly. But if you haven't got one, sort it out today. It's in week 1 of our Insider Programme, that's how essential it is.)

Now Google has made the process of removing a URL from the results a little easier, and Easier URL removals for site owners on Google Webmaster Central gives the full story. You'll still need – eventually – to properly indicate that the page no longer exists, or block the "Googlebot" from crawling the page, but you can fast-track its removal now.

Oops, I didn't mean to make that public

Are you on top of your 404s?

"Now when you visit the search queries feature in Webmaster Tools you'll see a new tab titled "Top Pages." The "Top Pages" tab lists impression, click, and position data for the top pages on your site based on their performance in Google's search results. If you click on one of the individual pages listed you’ll see a list of the queries driving traffic to that page, along with impressions and number of clicks for each query. Just like in the "Top queries" view, you can click on a specific query to see more detailed data and evaluate how the query is performing across the whole site."

I love this, and have just spent an hour finding out some intriguing data about one of my sites. And it won't be an hour wasted, because I think I can use the information to get a lot more traffic.

Why does one page do better than others?

Why would you block a search engine from visiting parts of your site? There are many reasons, such as parts of the site being duplicates of other parts, or sections being internal and private. Perhaps most important is the sad fact that the web is way too big for the search engines, and unless you're unfeasibly highly-ranked, they're not going to index everything you have to offer. So it makes sense to section off anything you don't really care about, and focus on directing the search engines to the pages you want people to visit.

If you can't think of anything you'd want to direct the search engines away from, then it might not be necessary to have a robots.txt file, although I like to include one anyway, because it just goes against the grain to serve up "page not found" codes, even to search engine robots. You can also give the search engines more detailed requests on a page level – see Using the Robots Meta Tag on the Google Webmaster Central Blog for more information. If you have a web designer on call, it might be worth asking if your robots.txt file is up to date and doing what you want. But also ensure they've read Serious Robots.txt Misuse and High Impact Solutions and don't go blocking off pages which are linked to externally, or make similar mistakes.

Protect your private parts