For our latest DeepCrawl webinar, we were very excited to welcome Emily Potter, Technical SEO Consultant at SEO A/B testing platform SearchPilot. Joining our host, DeepCrawl’s CAB Chairman, Jon Myers, Emily explained how SEO A/B tests are performed, the key insights she has learnt from running these tests, and why blindly following SEO best practice may actually be harming our websites.
We’d like to say a big thank you to Emily for her great presentation and for taking the time to share her knowledge and experience with us, as well as to Jon for hosting, and all those who attended.
You can watch the full recording here:
And review the slides here:
How SEO A/B Tests are run
SearchPilot deploys their SEO tests between the origin server and the CDN in order to make on-the-fly changes to a website, which allows them to run the tests for Google as well as users. Changes are being made to the source code rather than through client-side rendering to ensure the changes are made before they get to the user. This in turn means all of these changes are being indexed by Google, which enables SEO A/B testing to take place.
The best way to create a controlled testing environment is to have a large sample size of templated pages. Using a machine learning algorithm, the pages are split into two groups; the control template and the variant template. The changes to these groups of pages then get rolled out site-wide to measure how they perform against each other.
A positive test result will occur when the organic traffic of the variant group is larger than that of the control group, with a negative result seen when the control group performs better than the variant. An inconclusive test will therefore occur when there is no statistical difference between the two pages. The neural network model, which is being used to split up the pages, is also used when measuring the results.
SEO testing is complicated and there are a lot of external factors that need to be taken into account, for example, the SearchPilot model accounts for things such as;
- Seasonality
- Unexpected spikes in demand
- Algorithm updates
- Unrelated sitewide changes
- Competitor actions
Due to all of these external factors in play, it can be difficult to gain a statistically significant result.
From test results run up until 2019, SearchPilot were seeing 65% inconclusive tests and only 35% conclusive tests. Since Emily joined the team in 2019, they have seen a shift to 43% conclusive and 57% inconclusive. This, Emily explained, could be due to the different perspective she brings to the team, as they are seeing more evidence that old school SEO practice doesn’t always work. In addition, factors which were previously believed to not have an impact on SEO, are actually causing positive test results to occur.
Is best practice SEO actually best?
Emily explained that something the SearchPilot team have learned through their testing is that most ‘best practice’ SEO tactics fail. Emily explained that through their testing, the SearchPilot team have learned that most ‘best practice’ SEO tactics fail. Not only do they provide inconclusive results, but most are also actually harming organic traffic.
Meta Descriptions Example
One example Emily shared is optimising meta descriptions, while they no longer impact rankings, meta descriptions are still really important drivers of organic traffic to your site, where you advertise your page to the user in SERPs. However, Emily explained that rewriting, or optimising these, is not the best use of your time. This is because 91% of meta description tests they have run have had no detectable impact, as they were inconclusive. This is backed up by evidence we’re seeing more and more in search results, with Google generating snippets for several of the top-ranking pages, instead of using the meta description set.
In order to really test the impact of meta descriptions, Emily and her team decided to run some tests with the descriptions removed. The example Emily shared was for a nationwide pharmacy chain, where generic copy was written across many of the local pharmacy pages. After removing the meta descriptions and waiting for Google to reindex the pages, Google instead opted to show the opening hours and phone numbers for each local store. This is often what users are looking for when performing a local search but could lead to no click searches, which would not improve organic traffic. However, this test was actually positive and drove a statistically significant uplift in organic traffic.
This is an example of how SEO experience may be limiting you from some really good test ideas. An old school thought around meta descriptions may be ‘writing an enticing meta description will drive more clicks to your website’. However, a new school approach leans more towards satisfying user intent in order to drive more traffic to your site.
Swap best practice for a user first approach
Instead of blindly focusing on best practice approaches, Emily recommends leading with a user-first approach. While this may sound simple, users are actually really hard to predict and it’s even more complicated when you factor in the need to predict what Google thinks users prefer. This is why SEO testing is so important.
FAQ Schema Example
Emily shared an example of the importance of this by demonstrating the impact of adding FAQ schema. When adding structured markup to a website, you are only able to have one type displayed, for example, either price with star rating, or FAQ. While this may be positive, due to the FAQs taking up more real estate on the SERP, SearchPilot have found that for some websites, this has turned out to be negative. For some industries FAQ markup is proving to be more valuable than rating or price snippets, however for others, it is damaging their organic traffic.
Breadcrumb Schema Example
Another example Emily shared is breadcrumb schema, to transform how the URL appears for users in search results. While the general consensus is that breadcrumbs are valuable for driving organic traffic to a site, after testing the implementation of breadcrumb schema for a large eCommerce brand in the fashion industry, the results found were negative.
This is where the benefit of testing can really be seen, in order to determine which schema type is the most valuable for your site. By implementing this without testing it first, brands could see a decline in organic traffic without realising that it is due to this small change. These results further highlight the fact that big brands win by not making mistakes, and by testing things you can ensure these mistakes are not made.
One theory the team determined after reviewing the negative result of this test is the fact that, after adding the breadcrumb schema, the head term was actually removed from the displayed result. Despite this term still being in the title and meta description, it was important enough that losing it from the breadcrumbs harmed the organic traffic to the site.
So while belief around adding breadcrumb schema improves organic CTR, it’s still important to understand what users are looking for, and expecting to see, in search results in order to click through to the page.
This is why it is important that we don’t just blindly follow best practice approaches because we believe it is the right thing to do. We need to challenge these thoughts and instead think about what this means for users. Every website is different, as is the audience we are targeting. It’s also important to accept you don’t always know what the results will be, and this is why you need to be running tests, especially if you are a big brand.
Ranking factors need not apply
Based on some research Tom Capper undertook last year, he found that ‘for high volume queries, traditional ranking factors matter less in the top 5 results’. This means that the higher a page ranked, and the more competitive the keyword was, the less that things like backlinks and domain authority were actually influencing rankings. He found instead that it was user data impacting these pages. However, relying on user signals is difficult because we can’t predict how users are going to respond.
New Page Templates Example
An example of this was seen when implementing a new product page template, the test proved positive for CRO and Emily’s client wanted to make sure it didn’t harm the site’s organic traffic. This change involved using react to filter through different facets. However, as we know, JavaScript can cause a negative impact on SEO.
After running these changes as an SEO test, the result seen was positive and helped to improve their organic traffic. While no new content had been added, the small change had a significant positive impact on organic traffic.
There are two important points as to why this returned a positive result, firstly the JavaScript was server-side rendered, which means that Google can crawl the page without needing to load any JavaScript itself. Implementing react also led to a faster page load, which in turn helped to improve user experience.
Another example was for a client in the travel industry, who made a simple change of moving their search widget to the top of the page. While no content or keyword changes were made, the test actually displayed a negative result, which means this small amendment harmed their organic traffic.
Our old school best practice beliefs would lead us to believe that moving this element would not impact rankings, as no changes to content were made. However, it is clear that user signals can have a subsequent impact on SEO, and if these signals are negative they can harm organic traffic.
Some things don’t change
Of course, there are some SEO elements that are still just as important and one example of this is internal links. Internal linking tests are some of the most complicated that SearchPilot perform and Emily shared an example of this, with the same test run on two different sites.
This test involved adding some links between the main category and subcategories. In a typical A/B test you would only change the pages that are impacted, however when testing internal link changes, both the pages being linked from and to will be impacted.
When setting up these tests you would have the control group, which contains no new internal links, and the variant group which has the links added. The test then involved measuring what is happening on main category pages, as well as what happens to the pages receiving links and the combined effect of this.
The first example of this test was with a fashion eCommerce site, where a block of links was added to the top of the main category page. The results from this test showed that the main category page, where the links were added, was negative, while the sub-category pages receiving the links saw a positive impact. Therefore, the net result was neutral.
However, for a different industry, an online grocery site, adding the link buttons to the top of the main category page displayed a positive result for both the main and subcategory pages. Therefore, the test was positive overall and led to an increase in organic traffic.
When comparing the two tests to identify why these results were so different, SearchPilot looked at things including negative link equity, as well as potential keyword stuffing which could have happened on the first site. In comparison, adding the links to the second test example led to an increase in the number of keywords the site could rank for, as well as improving the relevancy for the page and creating new pathways for users.
Key Takeaways
- Don’t just blindly follow best practice approaches, think about the impact of these changes for the user and instead follow a user-centric approach to SEO
- Ranking factors need not apply for big brands, user signals are now influencing ranking more than ever before
- No two websites (or tests) are the same, it’s easy to follow best practice advice because that is what everyone else is doing, but testing for your individual site is invaluable.
Get started with DeepCrawl
If you’re interested in learning about how DeepCrawl can help you to identify technical issues on your site which are impacting both search engines and users, as well as benchmark data for running your own SEO A/B tests, why not get started with a DeepCrawl account today.
Discover DeepCrawl Automator
DeepCrawl Automator is a one of a kind product that helps to avoid traffic and revenue loss by mitigating the risk of introducing harmful code changes with new releases. Automator benefits the whole company by preventing errors from being released, speeding up development and QA processes and reducing the impact on production environments.
