The robots.txt file is one of the oldest and most elementary components of a website. We should all be aware of what we’ve got in ours!
“Action URLs” are web pages or windows that appear on our site when users are in some transactional process. They include shopping cart messages, or even just plain ‘thank you’ pages. Many of us have thousands of them without even realising it, because they’re ‘attached’ to content pages. What we don’t want is for the search engines to be wasting the site crawl time they give us, looking at these URLs. And the way we do this is through the robots.txt file.
Every site has this file (or at least should do), and it’ll be at [whatever.com]/robots.txt.
To tell the search engine crawlers not to look at certain URLs, their pattern just needs to be added to the robots.txt file. So on a shopping site, we might well see lines such as:
Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/
…or even more ambitiously, lines with wildcards:
Disallow: /*add-to-cart=*
The robots.txt file is one of the oldest and most elementary components of a website. We should all be aware of what we’ve got in ours!