Google Explains Discovery & Refresh Data in Crawl Stats Report

Google’s John Mueller provides extra element about new knowledge in Search Console’s up to date Crawl Stats report – the ‘discovery’ and ‘refresh metrics.

The Crawl Stats report in Google Search Console was up to date a number of weeks in the past and provides knowledge that wasn’t being reported on beforehand.

A particular part of knowledge, Crawl Purpose, got here up in the November 27 version of the Google Search Central dwell stream.

Mueller was requested to offer extra context on the 2 metrics included inside Crawl Purpose – proportion of ‘found’ URLs and proportion of ‘refreshed’ URLs.

Specifically, the next query was submitted:

“What’s the distinction between discovery and refresh? In our case it’s displaying 84% refresh.

Does that imply 84% of the time Google is crawling recognized URLs from their database, and solely 16% of the time they crawl our web site, sitemaps, and hyperlinks from different URLs from the recognized URL database?”

Google’s official Search Console assist doc provides transient descriptions of discovery and refresh:

• Discovery: The URL requested was by no means crawled by Google earlier than.
• Refresh: A recrawl of a recognized web page.

Mueller expands on that info in his response to the above query.

Mueller on ‘Crawl Purpose’ Data

Mueller prefaces his reply with disclose that he’s not 100% certain which URLs can be grouped into discovery and refresh, however he supplies his personal understanding of it.

Refreshed URLs seek advice from previously-crawled pages that have been crawled once more for the aim of updating the knowledge in Google’s search index.

Discovered URLs seek advice from pages on a web site that have been crawled for the primary time and by no means seen by Google earlier than.

Here’s how Mueller places it:

“I’m not 100% certain what precisely we’d put into every of these buckets, however usually we do break up issues up into refresh crawling the place we attempt to replace the knowledge that we’ve on a web site, and discovery crawling the place we attempt to discover new URLs that we’ve heard about from the web site. Which could possibly be issues like from new inner hyperlinks or from exterior hyperlinks pointing to your web site.”

Mueller provides {that a} refresh crawl includes updating content material whereas actively on the lookout for newly-placed hyperlinks.

“Refresh crawl doesn’t imply that we’re simply updating the web page’s content material, we’re additionally on the lookout for new hyperlinks which we are able to then use for locating new content material.”

When studying the Crawl Stats report web site homeowners ought to see a better proportion of refreshed URLs in comparison with found URLs.

Exceptions that come to thoughts are the launching of a brand new web site, migrating one web site with one other, importing a brand new sitemap, and different such actions.

If the report exhibits that quickly altering pages will not be being crawled usually sufficient, guarantee they’re included in a sitemap.

Pages that replace much less ceaselessly can be crawled much less usually, although web site homeowners can power a recrawl by manually pinging Google.

For the total query and reply from the Search Central stream seek advice from the video under. Full particulars about Google’s up to date Crawl Stats report may be discovered right here: Google Updates Search Console Crawl Stats Report.