Crawl Rate

Googlebot understands URL strucures well and can recognise faceted navigation and will slow down when it realises where the primary content is and where it has strayed from that. This is aided by GSC parameter handling.

Googlebot will crawl known URLs faster if it detects that significant changes have been made across the site to things like structured data, rel canonicals or redirects.

Images are crawled less frequently than content, so if these are migrated, to a CDN for example, it will take Googlebot longer to recrawl these.

If a site is being crawled too much or too little you can report a problem with Googlebot in the Help Center and this can be adjusted by a Google engineer.

When a site changes its hosting platform Google will be more conservative about how much they crawl as they don’t know how much load the new server set up will support.

Google takes no notice of the crawl frequency attribute in XML sitemaps or any priority set. Only the last modification timestamp will impact crawl rate.

Google uses a scheduler before crawling to work out when they need to recrawl URLs. Google will increase crawl rate if it gets signals that it needs to do so e.g. updated modification date in sitemaps and internal linking (especially from the homepage).

Setting your crawl rate to high means Google can crawl more but that doesn’t mean they will.

AMP pages may make log files look unusual as Google is requesting the page to update the cache but the page will appear to receive little traffic because users see the cached version of the page.

A page with a noindex tag will be crawled less frequently.