Today we're going to talk a bit about Screaming Frog SEO Spider, which is a powerful tool for both SEO specialists and website owners alike. It can show a sky high overview or a close up detailed inspection of a website. We will cover how to configure the tool to start a crawl and then how to start a basic analysis of the data.
How to use Screaming Frog to Audit Sites Quickly
If you just want to take a quick look at a prospect's website, you can use these settings. It will speed up the crawl if you skip the images, CSS, javascript and SWF when you're just trying to get a basic idea of what's going on with the website.
'Configuration' -> 'Spider'
"Crawl" tab -> Uncheck Resource Links boxes: Images, CSS, Javascript, SWF.
From here you can pick and choose if you want to check Page Links or even the XML sitemaps and shift the crawl behavior, too.
On the next tab, "Extraction", you can choose all the different kinds of details you want to look at, especially structured data like JSON-LD, RDFa, or Schema.org. This is a good way to see if the site is using structured data.
Moving on to the "Limits" tab, we can also choose more settings for how deep you want to go in the site crawl. Defaults are fine here, but the customization will allow you to lower settings for a faster crawl, or dive deeper with a longer crawl.
The "Rendering" tab has options to allow you to crawl sites that are not as crawler friendly. Default is text only, but you can shift this to Old AJAX or Javascript if necessary to crawl a site that's not working well with the crawler. If you notice you're not getting links during a crawl, try using these other rendering options.
Once you're happy with the spider configuration settings, you're ready to enter a URL to spider in the box at the top of the window, then click start on the right.
Just below where you entered your URL is a long row of tabs starting with "Internal". The cells below should start populating with all the pages that are linked to the URL you are crawling. You can click on each of these to get more information down below. The boxes on the right provide many different summaries.
How to Look at Screaming Frog Data in Spreadsheet
Go to the "Internal" tab at the top, just below the row of tabs, click the "Export" button. You can choose to build a comma separated value (.csv), .xls or .xlsx file that you can then open with excel or sheets.
Alternatively, I find it faster to just highlight the cells I want and copy (Ctrl+C), then paste (Ctrl+V) in your spreadsheet (if the paste has a weird format issue, then undo {Ctrl+Z} and then paste without formatting {Ctrl+Shift+V}). Only issue with this method is you won't have the header row.
How to see Screaming Frog Link Score with Crawl Analysis
Link Score is screaming frog's metric for internal links. After a crawl is complete, you can view your Link Score by clicking Crawl Analysis at the top of the program, then start. Once the analysis is complete you can scroll almost all the way to the right to the Link Score column. This is a good way to quickly analyze what pages need more internal links.
Esseodigital is a digital marketing service providing agency experience at a freelance price. If you’re interested in boosting your businesses marketing success, please contact us.
