Every website should have what’s known as a “sitemap”, as I’ve mentioned on many occasions. If you use a decent content management system, or if you used a decent web designer, this will have been taken care of. The sitemap is a text file with a list of all the pages on your site, usually in a text format called XML. It’s used by search engine crawlers to get a list of all the pages on your site without having to tortuously follow all the links and menus, although the crawlers will use that method too. As you add, delete and modify pages, the sitemap should get adjusted automatically. If you don’t have a sitemap*, you really should. There’s no reason why Google’s crawlers won’t find all of your pages by following through the links on your site, but why wouldn’t you provide all the help that you can?
But what about us poor human visitors? Do we need one too? Textbooks have been around for hundreds of years, and they’re a mature, efficient product. You wouldn’t think about publishing a textbook without a contents page or an index, so why wouldn’t a website have these? Yet most sites don’t.
I think that designers have developed substitutes for both, but there are deficiencies. In lieu of a contents page, there are cascading menu systems which flop open to reveal an overview of the site. This is OK, but the default position is to hide the data. And instead of a linear index of words and topics, there’s a search facility, which is very space-efficient, but always leaves users worrying that they might have searched incorrectly and missed something useful.
Indexing an entire site and keeping it maintained is a big task. But providing a contents page (or a “human readable sitemap”) should be a serious consideration. People like scanning lists. It’s how we’re brought up to look for information. It might be that you have to maintain this contents page manually, if your content management system can’t cope (and you may even prefer to do so, in order to give subjects the right priorities). But I find that once people have created a page like this, they find it presents information in a way which visitors find very comfortable.
* Usually the XML sitemap is called sitemap.xml, so you can find it at http://[your domain name]/sitemap.xml. If that comes up with nothing, that doesn’t mean you don’t have one; it might have another name. One way of telling the search engines its location is to use your site’s “robots.txt” file, so try http://[your domain name]/robots.txt and it might tell you there. Or you could try Google Webmaster Tools/Search Console (it’s under “Crawl”).