Crawling
Before a page can be indexed (and therefore appear within search results), it must first be crawled by search engine crawlers like Googlebot. There are many things to consider in order to get pages crawled and ensure they are adhering to the correct guidelines. These are covered within our SEO Office Hours notes, as well as further research and recommendations.
For more SEO knowledge on crawling and to optimize your site’s crawlability, check out Lumar’s additional resources:
Ensure Google is Able to Crawl All Resources Required for Mobile Friendliness
When testing sites for mobile friendliness, Google needs to render the page and will be unable to confirm if it is mobile friendly if it cannot access all of the content and resources required. The Mobile Friendly Testing tool can be used to check if there are any resources required for mobile friendliness that are blocked e.g. CSS or JavaScript.
Crawl Budget Not Affected by Response Time of Third Party Tags
For Google, crawl budget is determined by how many pages and resources they fetch from a website per day. If a page has a large response time they may crawl the site less to avoid overloading the server, but this will not be affected by any third party tags on the page.
Evergreen Googlebot Has Fully Rolled Out
The new evergreen Googlebot has been in testing for several months and was crawling 10% of the web last November, but is now 100% live. The URL inspection and Mobile-friendly testing tools are being updated to reflect this and will be launched soon.
Chrome 74 Has a Slight Impact on Google’s Ability to Trigger Infinite Scroll
Due to new features now being supported in Chrome 74 (for example intersection observer), infinite scroll can now be used without a polyfill. However, Martin recommends retaining the polyfill until the testing tools are updated, so these changes can be thoroughly tested.
Using Chrome 41 Won’t Simulate an Exact Googlebot View
You will see results similar to what Googlebot sees by using Chrome 41, but not exactly the same. John wouldn’t recommend using old versions of Chrome to browse due to the security issues. The Mobile Friendly Test or URL Preview in GSC are more accurate in displaying how Google views a page.
Speed Metrics Important for UX Are Different to Metrics Important for Crawling & Indexing
While there is some overlap, speed metrics that are important for UX are completely different to those used for crawling and indexing. For the latter, Google needs to be able to request the HTML pages as quickly as possible and server response time is also important.
Check Server Logs If More Pages Crawled Than Expected
If Googlebot is crawling many more pages than it actually needs to be crawled on the site, John recommends checking the server logs to determine exactly which pages Googlebot is crawling. For example, it could be that JavaScript files with a session ID attached are being crawled and bloating the total no. crawled pages.
Fluctuations in Googlebot’s Download Time Caused by Changes In Server Speed, Page Size and Page Complexity
If GSC is showing that Googlebot is spending more time downloading pages on a site, the pages may have become large, more complicated or it could mean that the server is slower.
Make Sure Old Site Isn’t Being Blocked from Crawling During Migration
During a site migration, if the old site is being blocked from being crawled by Google in robots.txt, this can cause issues with Google being able to process the migration and pass on signals to the new site.
New Googlebot User Agents Are Updated Within the Official Google Documentation
Googlebot user agent strings are listed in the official Google documentation and are updated fairly frequently, so if you’re getting visits that are similar to Googlebot but aren’t in the documentation, then they are likely to be fake or a user running cloud hosting. Run a reverse lookup for Google’s IP address to double check.