Hidden Content

An attendee asked whether a website should disallow subdomains that are sending API requests, as they seemed to be taking up a lot of crawl budget. They also asked how API endpoints are discovered or used by Google.

John first clarified that API endpoints are normally used by JavaScript on a website. When Google renders the page, it will try to load the content served by the API and use it for rendering the page. It might be hard for Google to cache the API results, depending on your API and JavaScript set-up — which means Google may crawl a lot of the API requests to get a rendered version of your page for indexing. 

You could help avoid crawl budget issues here by making sure the API results are cached well and don’t contain timestamps in the URL. If you don’t care about the content being returned to Google, you could block the API subdomains from being crawled, but you should test this out first to make sure it doesn’t stop critical content from being rendered. 

John suggested making a test page that doesn’t crawl the API, or uses a broken URL for it,  and see how the page renders in the browser (and for Google).

Interstitials (for example, those that require users to verify their age before browsing) can have a negative impact on crawling and indexing if not implemented properly. Googlebot doesn’t click buttons or fill in forms, so any interstitial that requires those actions before the page is loaded may prevent Googlebot from crawling the content itself. Ideally, it’s recommended to use JS/CSS to display interstitials on top of existing content that’s already been loaded in. Because the content is still being loaded and users can access it after navigating the interstitial, this would not be seen as cloaking.

One user asked whether all hidden text goes against Google’s Webmaster guidelines. John explained that hidden text becomes problematic when it’s there to deceive search engines. Google is generally pretty good at identifying when this is the case, so hidden text that’s not deceptive is generally not a problem. One example given is around accessibility. Content that’s there to aid screen readers (but isn’t visible on the page) is just one way that hidden text can serve a very valid purpose.

John was asked about a website that had a lot of noindexed pages that had HTTP errors. They asked whether it’s considered ‘cloaking‘ to show an empty HTML page to bots to get those URLs de-indexed, while still showing users the page.

John mentioned that the part of ‘cloaking’ that is an issue is when search engines get more or vastly different content than users. Google wants to avoid promising users something they can’t find when they go to a page from a query. However, showing an empty page with a noindex will cause Google to drop those URLs and they will not care if users see something different because the page will not appear in search results.

A question was asked about whether using noscript tags could be a workaround for getting content seen by Google. John said that Google generally ignores content in noscript tags, so it wouldn’t be a workaround if the content you are including in noscript was content that you wanted to be included for indexing.

In response to a question about whether there are cloaking issues around showing Google different content vs. what a user would see on a more personalized page, John clarified that only what Googlebot sees is indexed. Googlebot usually crawls from the US and crawls without cookies, so whatever content is there would be what is indexed for the website. So, on personalized pages, make sure that you’re only changing things for users that are not critical to how you want to be seen in search.

If you have content hidden behind a tab or accordion, John recommends using the view source or inspect element tool to ensure the content is in the HTML by default. Content pre-loaded on the HTML will be treated as normal content on the page, however, if it requires an interaction to load, Google will not be able to crawl or index it.

An accordion style format can be used with FAQ structured markup to expand the answers when clicked on, as long as the question is visible by default.

Google doesn’t necessarily prioritize the content above the fold when ranking a page, but will also take into account other elements on the site.

If you have hidden content which you want to be indexed, ensure it is implemented using CSS, rather than sever-side JavaScript, to enable Google to see the content has been loaded when they crawl and render the page.