Google Search Console Tips

One user asked about having hundreds of pages showing as ‘Indexed, though blocked by robots.txt’ in GSC. This only really becomes a problem if the blocked pages are ranking in place of the content you want indexed. Much of the time, pages showing this error in GSC can only be found via a site: search, and even then many of them get omitted from the initial results. It’s highly unlikely that users would ever come across these, so digging into how and why they’re being found by other sources becomes a much lower priority. If they are showing up in place of actual content, you need to question why Google isn’t prioritizing the desired version in the way you’d expect them to.

A user asked about the difference between PageSpeed Insights (PSI) scores and the CrUX (Chrome User Experience Report) data found in Google Search Console (GSC). According to John, ultimately there’s no ‘correct’ number to use when it comes to speed and performance assessments — both PSI and GSC data can be used for different things. 

The PageSpeed Insights score calculates a simplified overall score based on a number of assumptions about your website’s speed and the average user. Google Search Console, on the other hand, uses Core Web Vitals data and a sample of actual visitor experiences relating to speed, responsiveness, and interactivity.

John points out that there’s a difference between “field data” — numbers based on what real users have seen when they go to your website — and “lab data”, which is more of a theoretical view of your website, based on more generalized assumptions about the average user (with considerations for which devices most people will use, average internet speeds, etc.).  

Both types of data are designed to give you insights into what users are experiencing or are likely to experience on your site. 

John recommends using the field data in GSC to understand the current situation on your website and the lab data (namely individual tests you can run yourself) to optimize your site and try to improve things for the future. When you’re happy with the lab data you’re getting after making updates to your site, over time you will also automatically be collecting the field data in GSC. You can double-check the field data against the lab data to ensure that actual users are experiencing the improvements. 

One interesting question was why the Google search results of a site:query don’t match what Search Console shows for the same website. John responded that there are slightly different optimizations for site:query.

When site:query is used in Google search to determine the number of indexed URLs, Google just wants to return a number as quickly as possible and this can be a very rough approximation. If you need an exact number of URLs that are indexed, he clarified that you should use Search Console to get this information. GSC is where Google provides the numbers as directly and clearly as possible. These can fluctuate, but overall the number shown in Search Console for the indexing report is the number of URLs you have indexed for a website — and is likely to be more accurate than the site:query results shown in the SERPs.

The site:query result is only a rough approximation of pages indexed

The titles shown in search results on Google are based on the page itself and not the search query. If you notice that the title displayed in the SERPs is being changed between older and newer versions of a page, it could be that an older version of the page has been used for indexing and processing. You can resubmit that page for reprocessing and once it’s processed, use a site:query to recheck the title. You could use this process to understand how Google is changing it and to help to fine-tune your pages (and even understand changes that may be needed for larger templates).

Users may see a difference in schema validation across tools. This is because the schema.org test is designed to validate all theoretical schemas, whereas the reports in Google Search Console and the Rich Results Test focus only on schema that can have a visible effect on Google search results. It could also be a simple case of the requirements being different across both tools; Google tends to be a little stricter on its requirements than schema.org. If the goal is simply for the marked-up content to appear in Google’s search features, it’s best to follow Google’s own guidelines.

A site owner was mentioning to John that they saw a drop in crawl rate which was attributed to high response times. They’ve since fixed this, but the crawl rate has not changed. John explained that it may just need longer to catch up, but there is also a form within the Google Search Console Help Center where you can request someone to take a look at the crawling of your website—they might be able to adjust it manually.

Continuous scrolling in Google search results is being rolled out on mobile English-language results in the US. For now, it will work on the first four pages only (that is the top 40 results). Users will need to manually click a ‘load more’-style button in order to see URLs that rank in positions after page 4. 

Google’s new “continuous scroll” feature for its SERPs will continue to load results in sets of 10, so impressions in GSC will continue to be counted in the same way. For example, when a user scrolls down and organic results 11-20 load dynamically in the viewport, those pages will get an impression in the same way as if the user had clicked through to page 2.

If anything, site owners might see a slight increase in impressions as continuous scroll makes it easier for users to view more results, although this could in turn have a negative impact on CTR.

Google filters and blocks bots at different positions in their search systems. It’s still sometimes possible to see impressions from bots and scrapers appearing as impressions in Google Search Console (GSC). If bots appear in your GSC, you can flag these in the ‘feedback’ function. Sometimes bots are filtered out at a later stage in the search system but still will appear in GSC. 

If Google’s testing tools like the Mobile Friendly Test, won’t work with a disallowed or noindexed locally hosted development site, Google provides a guide to implementing a proxy to bypass any firewalls.