Crawl Rate

If you move your site to a CDN, Google sees it as a new hosting setup and may temporarily reduce the crawl rate.

Parameters in Search Console aren’t a directive, more of a hint to Google not to crawl these as frequently. Blocking URLs in Robots.txt is strong directive to say Google should not be looking at these pages e.g. if Google’s crawls are crashing your site.

Submit a sitemap file with an update last modification date to speed up the process of crawling and indexing of pages that have been changed.

Google recommends keeping redirects for one year minimum. This is because URLs will, in the worst case scenario, be crawled by Googlebot every six months, so a having the redirect in place for a year would give Google the chance to see that redirect at least twice.

This is done because Google doesn’t want to spend a long time accessing slow to load pages and so as not to put more of a load on an already struggling server.

Updating of HTML Improvements in Search Console is dependent on crawl frequency of the website. John recommends using this report to get an idea of what you could be working on your site but not as a checklist you need to get down to zero issues.

If you change the IP address, Google recognises a change in the infrastucture and will start to crawl cautiously at first, then increase the crawl rate over time, but it won’t affect rankings.

Google crawls site-wide changes a bit faster than they normally would.

Google’s crawling systems will recognize an increase in server errors (5XX errors) and assume they’re crawling too hard and will step back appropriately. If the URLs returning the errors are not valid pages, consider returning a 4xx status instead.

Google limits the crawl rate for sites on the same server so that it doesn’t overload the server when crawling these sites.