Crawl Rate

Google tries to crawl higher level pages every day, lower level pages may take a few weeks or months, and the content is processed within a day. The impact of context changes on rankings is delayed and takes a bit of time.

Pages with more internal links are more likely to be crawled more frequently.

It makes sense to submit both the old and new URLs when you have changed them. Google may increase the crawl rate automatically when it thinks something has changed.

Landing Pages which aren’t linked from the main site are not spam but may take longer to be crawled, and without links, Google will have trouble understanding the context, so John recommends they should be linked internally.

If you experience a temporarily high crawl rate, it might be caused because of an increase in authority, or Google thinks the server can handle an increased load, or they might be finding a lot of duplicate content caused by things like URL parameters.

If you move to a new ‘server infrastructure’, Google may reduce the crawl rate until it has confidence that it can crawl at a faster rate.

If content is buried deep in the site, it might take longer for Google to discover it, or changes. so improving internal linking from higher levels will help pages be crawled faster.

Crawl rate is somewhere between minimum list of pages Google wants to update, and the maximum number of pages they think it’s safe to crawl without impacting performance. Any new pages discovered can be crawled provided there is some remaining budget, but might get queued up for the next day.

Assuming Google is able to pick up your content, there’s no ranking benefit for a page being crawled more frequently.

PDFs won’t be crawled as often as HTML pages because the content is generally more stable.