Crawl Rate

A site owner was mentioning to John that they saw a drop in crawl rate which was attributed to high response times. They’ve since fixed this, but the crawl rate has not changed. John explained that it may just need longer to catch up, but there is also a form within the Google Search Console Help Center where you can request someone to take a look at the crawling of your website—they might be able to adjust it manually.

If Google is spending more time crawling a particular group of slow pages then it may make the average fetch time and crawled data look worse.

The resources that Google fetches when they render a page are included in Google’s crawling budget and reported in the Crawl Stats data in Search Console.

If Google can’t recrawl pages quickly enough due to poor page response times then they won’t recrawl pages as often as they would like to.

Google’s crawl stats in Search Console are an accurate reflections of Google’s own crawling logs, but they include URLs fetched from other Google services which use the same infrastructure as Googlebot, including Google Ads landing page checks and product search crawling.

Google tends to recrawl pages at least once every six months as an upper limit.

When determining the crawl rate for a website, Google will try to adjust it automatically over time, taking into account how fast the site returns results and how much content there is to crawl.

Google is able to detect how frequently the different sections of a site are updated and crawl them at different speeds, so that the frequently changing pages are crawled more regularly.

If GSC is showing that Googlebot is spending more time downloading pages on a site, the pages may have become large, more complicated or it could mean that the server is slower.

You can’t instruct Google to crawl pages less frequently, but instead you can put more emphasis on more important pages to be crawled by using internal linking.