Deepcrawl is now Lumar. Read more.
DeepcrawlはLumarになりました。 詳細はこちら

The Zen Guide to HTTPS Configuration

SEO and Digital Marketing Best Practices

Unless you’ve been living under a rock since August 2014, you’ll have heard that Google has started using HTTPS as a lightweight ranking signal, and they say they have plans to strengthen the signal in the future.

Essentially this means that sites using HTTPS (secure, encrypted connections) may be ranked slightly higher than those using just HTTP.

Although Google assures us that this is only a lightweight signal for the time being, and affects fewer than 1% of global queries, they also know that this announcement is enough to panic people and trigger a mad rush to HTTPS.

It also marks the start of a big change in the way that sites will be built and managed in the future. Traditionally, HTTPS has only been beneficial on sites that collect users’ personal data such as credit card details or email addresses, whereas now HTTPS configuration will be of benefit to all sites, even content-only sites and blogs.
 

Step 1: Don’t panic

First things first: this is not a major ranking signal at the moment and it may never be significant. In the long-term, Google say they ‘may decide to strengthen’ the weight of this signal, but it still currently carries less weight than others like high quality content, and only affects a tiny number of queries.

Add to that the cost, development and user experience implications of migrating or making changes to a domain: this is not a matter to be rushed. So let’s calm ourselves and start with a few simple checks…
 

Step 2: Check your current configuration

  • Verify both HTTP and HTTPS in Search Console: This will be essential for setup and migration to HTTPS.
  • Check if your HTTP or HTTPS domain redirects to the other, and which HTTP status code is used, with a HTTP checking tool like Fetch as Lumar (formerly Deepcrawl), Fetch as Googlebot, or Web Sniffer.
  • If the domains don’t redirect, check the canonical tags: The canonical tag should always include HTTPS in the URL, even when looking at the HTTP URL.
  • Check if your site has HTTPS pages indexed in Google by doing a site: command to check the number of pages on HTTP or HTTPS. To check for HTTP, you’ll need to enter ‘site:example.com -inurl:https’.
  • Check your SSL certificates are valid: We recommend SSL Shopper’s SSL checker.
  • Check if mixed HTTP/HTTPS: Again, SSL Shopper’s Insecure Sources checker is good for this.
  • Check if your ads can be served on your HTTPS site.
  • Sign up for a free Robotto account, which will alert you when your HTTP/HTTPS configuration changes.

 

Step 3: Choose your preferred hosting option

Once you’ve understood your current configuration, you can decide which setup you want and your next steps.
 

Option A: Host on both, canonicalise to HTTPS

This method means you can maintain HTTP URLs for other uses, but Google will index and direct traffic to your HTTPS pages. You will essentially be running two different domains (one using HTTP, another using HTTPS) but Google will only index the HTTPS version.

Although this seems like the simplest option since you get to keep your HTTP URLs, it does come with some potential complications:

  • Since you will be running two domains, it’s imperative that your canonical tags work properly to avoid duplicate content issues.
  • You will still need to migrate all your URLs in Google’s index to HTTPS versions. The first step in this migration is to update your canonical tags to always use the HTTPS URLs.
  • Users coming from any source other than search engines won’t get the benefits of a HTTPS site. Even users of content-only sites are still better protected by a secure HTTPS site than a HTTP one, since the extra security ensures your content cannot be modified or corrupted on its way to the reader. For more information, see the Google I/O 2014 – HTTPS Everywhere video.

For more information on using canonical URLs for HTTPS, read this Google support post.
 

Option B: Host on HTTPS, redirect HTTP

This is the option that Google recommends. A 301 redirect means that all users – whether coming through search engines or otherwise – will get all the encryption, data integrity and authentication benefits of a secure HTTPS site. Provided the HTTP/HTTPS is the only thing that changes in the URLs – https://example.com/url is redirected to https://example.com/url – traffic shouldn’t be affected.

Since there is no way to manage HTTP/HTTPS in Search Console, this option means your site will be using (and indexed as) a different domain.

However, migrating to a different domain could involve a hefty amount of development work, and needs exhaustive research, management and testing to ensure nothing goes wrong. Your site will have to be modified to work with HTTPS, and in some cases this won’t be possible, meaning it will have to be completely redeveloped.

Here’s a very basic process for migrating to HTTPS:

  1. Prepare site for operation on HTTPS (potentially massive amount of development work).
  2. Verify both versions in Search Console. You don’t need to use the change of address tool, as this is only for moving between different domains.
  3. Use a 301 redirect to direct traffic and search engines from HTTP to HTTPS.

While some Googlers are pressing for sites to move to HTTPS, John Mueller has discussed the unexpected problems that migrating has caused and Google lists the problems that can occur on its support section. This suggests you shouldn’t go ahead until all potential problems have been ironed out and you’re satisfied that your own configuration is safe on HTTPS.

Include the following considerations in your migration plan, plus any others that are particular to your site:

  • Don’t use 302 redirects: they’re not a clear signal that the site has moved to HTTPS. Google specifically state that 301 redirects should be used.
  • All resources (JS, CSS, images) need to be on HTTPS. For more information see Maile Ohye’s SMX Advanced 2014 presentation.
  • Internal links, Sitemaps, canonical tags, robots.txt file and analytics tracking codes need to be updated to refer to HTTPS.
  • Third-party resources (eg. adverts, some RSS services) might not be set up to work with HTTPS. Check this before deciding whether to migrate.
  • Similarly, if you provide any widgets or resources to another third-party and they are on a different HTTP/HTTPS version, you might find that there are clashes between the security set-up and you might encounter problems.
  • Analytics and backlink data could be affected, especially if many sites in the same industry have switched to HTTPS. You could try to get your backlinks updated to use the HTTPS URL, but this is a load of work and probably pointless; the backlinks will still carry the same weight in search engines with the addition of a 301 or canonical tags.
  • Social shares also need to be migrated/managed to retain social proof (only Facebook, Google +1 and LinkedIn shares transfer automatically, although this can still take weeks/months). Search Engine Watch has a comprehensive guide to maintaining social shares after a site migration.

 

Extra advice on migrating

  • Once you’ve migrated your site and 301 redirected everything to the new version, John Mueller suggests uploading your disavow file again to the new version. This will avoid any negative effects caused by passing pagerank from the old site to a new one that doesn’t have a disavow file.
  • Don’t use the use the URL removal tool to resolve problems with URLs on the wrong protocol, as removing one version will also remove all www/non-www and http/https versions of the URL.
  • Google recommend using a web server that supports HTTP Strict Transport Security (HSTS), which will tell the browser to request pages using HTTPS automatically, and tells Google to serve secure URLs in the search results.
  • Both Search Console / Bing Webmaster Tools need correct settings applied: verify ownership of both HTTP and HTTPS versions, but only leave one available.

 

The problem with HTTPS: issues to bear in mind

 

HTTPS loads slower than HTTP

HTTPS pages may be slightly slower than HTTP, so engagement data might be affected. However, from a purely SEO point of view, the HTTPS ranking signal may compensate for this. Use best practice to limit the impact, and consider using a web server that supports HTTP Strict Transport Security (HSTS), which could also help with speed issues on HTTPS.
 

This kills referral tracking between websites

Similar to the effect that Google’s move to secure search had on organic keyword data, the long-term impact on visibility when a lot of websites move to HTTPS domains could be huge and we won’t be able to assess backlinks on their traffic-driving potential.
 

Poor HTTP/HTTPS URL support in Google Analytics and Search Console

There is currently no way to natively distinguish HTTPS traffic in Google Analytics, and Search Console treats HTTPS as a different domain.

You can’t currently specify your preferred configuration in the same way you can for www/non-www domain variations in Search Console; Google just recommend verifying both HTTP/HTTPS versions in Search Console and then 301 redirecting to HTTPS.

The unique data in some of the features of Search Console can be used the monitor your migration. For example, when switching to HTTPS, you should see the data reduced in features like index status and search queries on the HTTP version, but increasing on the HTTPS version. John Mueller discussed this in a recent Webmaster Tools hangout.
 

HTTPS and HTTP/2

 

What is HTTP/2?

HTTP 1 was designed for webpages with few external assets. Browsers typically downloaded assets sequentially, but this wasn’t a problem on lighter pages. Now most webpages have 50+ resources, which is difficult for HTTP 1 to handle.

HTTP/2 downloads many resources at the same time, prioritizes them and supports compressed HTTP headers.
 

HTTP/2 and Google

Chrome will move to HTTP/2 by 2016 and servers must be upgraded in order for a site to reap the benefits.

As far as we know there’s no SEO benefit to switching, but the fact that Google is pushing it – and that it has to do with speed and user engagement – implies that there could be in the future. Either way, any increase in site speed should help engagement, which in turn should have an indirect SEO benefit.
 

HTTP/2 and HTTPS

Firefox have stated that they have stated that they will only support HTTP/2 over TLS, which would make HTTPS mandatory. Other browsers may follow suit, which would further increase the benefits of moving to HTTPS.
 

Testing HTTPS with Lumar (formerly Deepcrawl): useful settings and reports

 

Choose HTTP/HTTPS when setting up a project

Lumar will start on the the version you specify, but crawl both if they are interlinked, and report on the configuration in terms of redirects and canonical tags.

HTTP & HTTPS Configurations
 

Add the alternative as an alias

Add the alternative to the above as a domain alias (under Advanced Settings) to force Lumar to crawl both HTTP and HTTPS: this will allow you to see your configuration.

DeepCrawl - Crawl HTTP/HTTPS

Our other tool, Robotto, will allow you to monitor any changes in your set-up.
 

Check canonical tags work properly (for option A above):

You can check canonical tags work from one to the other using Lumar’s Orphaned Canonical Pages report.

DeepCrawl Orphaned Canonicals report
 

Grab a list of all redirected links

The Validation > All Redirected Links report will show all links to pages that return 301/302 redirect status or include meta refresh redirect.

Use this to download a full list of the internal links that need to be updated to https version.
 

Identify domain duplication

Lumar provides a report that includes all domain duplication variations and will identify any full site duplication.
 

Find internal broken links

Use the Validation > Internal Broken Links report to update any links that have been broken thanks to any errors during the migration.
 

Find 302 redirects

Find 302 redirects via the Indexation > Non-301 Redirects report and get them updated to 301s.

Avatar image for Tristan Pirouz
Tristan Pirouz

Marketing Strategist

Tristan is an SEO enthusiast, strategist, and the former Head of Marketing at Lumar.

Newsletter

Get the best digital marketing & SEO insights, straight to your inbox