Skip to content

Five things you can check with a site crawl

Having a good overview of any websites we manage is essential. I’ll admit to having let many sites get out of control in the past, with far too many old pages containing ignored SEO problems.

I’d bet most website managers, if asked how many pages there are on their sites, would guess way too low. If asked how many files there are in total, including images and PDFs, they’d be even further out on the low side.

The ones who are on top of things are the ones who crawl their sites frequently with a website ‘spider’. The standard industry application for this is the Screaming Frog SEO Spider, which is a quite brilliant tool, but it does cost £149 a year. There’s a free version which will handle up to 500 URLs, but as suggested above, many of us who think our sites come in at under that level will be sadly mistaken. However, there’s no cost to give it a try.

What are the main things to look for? I’d say there are five:

  1. Broken links – both internal and external
  2. Indexability – ensuring all URLs return a 200 status
  3. Titles and meta data – confirming they’re complete and conform to best practice
  4. Checking for forgotten old content
  5. Checking for duplicate pages

Beyond this, there’s a lot more that Screaming Frog can do, including the ability to compare crawls. It’s almost certainly the best SEO tool any of us can invest in.

Note: BMON Google Ads clients can request a crawl of their site, done by us, at no charge.