5 Top Crawl Stats Insights in Google Search Console

There is one report in Google Search Console that’s each insanely helpful and fairly laborious to seek out, particularly if you happen to’re simply beginning your search engine optimization journey.

It’s probably the most powerful tools for each search engine optimization skilled, regardless that you may’t even entry it from inside Google Search Console’s essential interface.

I’m speaking concerning the Crawl stats report.

In this text, you’ll study why this report is so essential, how you can entry it, and how you can use it for search engine optimization benefit.

How Is Your Website Crawled?

Crawl budget (the variety of pages Googlebot can and needs to crawl) is important for search engine optimization, particularly for big web sites.

If you could have points along with your web site’s crawl finances, Google could not index a few of your useful pages.

And because the saying goes, if Google didn’t index one thing, then it doesn’t exist.

Google Search Console can present you what number of pages in your web site are visited by Googlebot on daily basis.

Armed with this data, yow will discover anomalies which may be inflicting your search engine optimization points.

Diving Into Your Crawl Stats: 5 Key Insights

To entry your Crawl stats report, log in to your Google Search Console account and navigate to Settings > Crawl stats.

Here are all the knowledge dimensions you may examine contained in the Crawl stats report:

1. Host

Imagine you could have an ecommerce store on store.web site.com and a weblog on weblog.web site.com.

Using the Crawl stats report, you may simply see the crawl stats associated to every subdomain of your web site.

Unfortunately, this methodology doesn’t presently work with subfolders.

2. HTTP Status

One different use case for the Crawl stats report is trying on the standing codes of crawled URLs.

That’s since you don’t need Googlebot to spend sources crawling pages that aren’t HTTP 200 OK. It’s a waste of your crawl finances.

To see the breakdown of the crawled URLs per status code, go to Settings > Crawl Stats > Crawl requests breakdown.

In this specific case, 16% of all requests have been made for redirected pages.

If you see statistics like these, I like to recommend additional investigating and in search of redirect hops and different potential points.

In my opinion, one of many worst instances you may see right here is a considerable amount of 5xx errors.

To quote Google’s documentation: “If the positioning slows down or responds with server errors, the restrict goes down and Googlebot crawls much less.”

If you’re in this subject, Roger Montti wrote a detailed article on 5xx errors in Google Search Console.

3. Purpose

The Crawl stats report breaks down the crawling objective into two classes:

• URLs crawled for Refresh functions (a recrawl of already identified pages, e.g., Googlebot is visiting your homepage to find new hyperlinks and content material).
• URLs crawled for Discovery functions (URLs that have been crawled for the primary time).

This breakdown is insanely helpful, and right here’s an instance:

I just lately encountered a web site with ~1 million pages categorized as “Discovered – presently not listed.”

This subject was reported for 90% of all of the pages on that web site.

(If you’re not conversant in it, “Discovered however not index” implies that Google found a given web page however didn’t go to it. If you found a brand new restaurant in your city however didn’t give it a attempt, for instance.)

One of the choices was to attend, hoping for Google to index these pages step by step.

Another choice was to take a look at the information and diagnose the difficulty.

So I logged in to Google Search Console and navigated to Settings > Crawl Stats > Crawl Requests: HTML.

It turned out that, on common, Google was visiting solely 7460 pages on that web site per day.

But right here’s one thing much more essential.

Thanks to the Crawl stats report, I discovered that solely 35% of those 7460 URLs have been crawled for discovery causes.

That’s simply 2611 new pages found by Google per day.

2611 out of over 1,000,000.

It would take 382 days for Google to completely index the entire web site at that tempo.

Finding this out was a gamechanger. All different search optimizations have been postponed as we totally centered on crawl finances optimization.

4. File Type

GSC Crawl stats may be useful for JavaScript web sites. You can simply verify how incessantly Googlebot crawls JS information which can be required for correct rendering.

If your web site is full of pictures and picture search is essential on your search engine optimization technique, this report will assist lots as nicely – you may see how nicely Googlebot can crawl your pictures.

5. Googlebot Type

Finally, the Crawl stats report provides you an in depth breakdown of the Googlebot kind used to crawl your web site.

You can discover out the proportion of requests made by both Mobile or Desktop Googlebot and Image, Video, and Ads bots.

Other Useful Information

It’s value noting that the Crawl stats report has invaluable data that you simply gained’t discover in your server logs:

1. DNS errors.
2. Page timeouts.
3. Host points similar to issues fetching the robots.txt file.

Using Crawl Stats in the URL Inspection Tool

You may also entry some granular crawl knowledge exterior of the Crawl stats report, in the URL Inspection Tool.

I just lately labored with a big ecommerce web site and, after some preliminary analyses, seen two urgent points:

1. Many product pages weren’t listed in Google.
2. There was no inner linking between merchandise. The solely approach for Google to find new content material was via sitemaps and paginated class pages.

A pure subsequent step was to entry server logs and verify if Google had crawled the paginated class pages.

But having access to server logs is commonly actually tough, particularly whenever you’re working with a big group.

Google Search Console’s Crawl stats report got here to the rescue.

Let me information you thru the method I used and that you need to use if you happen to’re scuffling with an identical subject:

1. First, lookup a URL in the URL Inspection Tool. I selected one of many paginated pages from one of many essential classes of the positioning.

2. Then, navigate to the Coverage > Crawl report.

In this case, the URL was final crawled three months in the past.

Keep in thoughts that this was one of many essential class pages of the web site that hadn’t been crawled for over three months!

I went deeper and checked a pattern of different class pages.

It turned out that Googlebot by no means visited many essential class pages. Many of them are nonetheless unknown to Google.

I don’t suppose I want to clarify how essential it’s to have that data whenever you’re engaged on enhancing any web site’s visibility.

The Crawl stats report permits you to look issues like this up inside minutes.

Wrapping Up

As you may see, the Crawl stats report is a strong search engine optimization software regardless that you possibly can use Google Search Console for years with out ever discovering it.

It will assist you diagnose indexing points and optimize your crawl finances in order that Google can discover and index your useful content material shortly, which is especially essential for big websites.

I gave you a few use instances to think about, however now the ball is in your courtroom.

How will you utilize this knowledge to enhance your web site’s visibility?

More Resources:

---

Image Credits

All screenshots taken by writer, April 2021