Sitecore 8: Test everything, everywhere. 

On the first day of their 2014 Symposium, Sitecore introduced their new product: Sitecore 8. While nonlinear’s 6 Sitecore MVPs have had plenty of early looks at the new release, even we were surprised by how central the concept of testing has become to the Sitecore roadmap. It’s their attempt to radically change how authors think about the content they publish – and the value they deliver for their companies.

Testing is now at the heart of content publishing in Sitecore

We’ve argued elsewhere that A/B testing is the most underused/highest value digital marketing activity. Very few organizations regularly test their content in the real world to determine what drives positive results and what reduces website effectiveness.

Sitecore Symposium 2014

Sitecore has supported A/B and multivariate testing for several releases, but with Sitecore 8, they make it absolutely central to the content creation and publication process. Sitecore 8 introduces three big advances over versions 7.x:

1. It makes it exceptionally easy for non-experts to optimize content: testing with one click

The standard optimization interface aims to make testing content as simple as publishing content. It hides the required optimization complexity within a sleek streamlined wizard. They even introduced gamification elements, encouraging content publishers to predict which version would win. (There is also an interface designed specifically for the optimization expert with all the bells and whistles that compares favourably to other products in the market.)

2. Fine-grained definition of goals against which to test

One of the more frustrating aspects of the 7.x version of Sitecore’s Digital Marketing System was the inability to easily evaluate an A/B test's effect on a specific outcome – the system insisted on testing against “Engagement Value,” a concept that many found nebulous and which rarely mapped to the real world needs of marketers. We've got to give Sitecore credit for listening to their customers; Sitecore 8 allows marketers the freedom to define evaluation criteria, while still making it clear how the test effects the overall visitor experience.

3. Visualization of A/B tests by persona

As we have discussed elsewhere, sometimes A/B tests lie: Sitecore 8 makes it easy to visualize how a test impacts different types of visitors. For example, you can gain insight into whether a specific example improves results for “soccer moms” but suppresses results for “sports car enthusiasts.”

Sitecore 8

"We envision a world where everything is tested" Michael Seifret

Brilliant vision, big challenges

Michael Seifert, Sitecore CEO, said many smart things during his keynote speech, but he missed the mark when he said, “Why don’t we test every change we make online? Because there is no simple tool for testing anything.”

He’s correct that simple tools have been missing, but this is not the only barrier marketers face. It may not, in fact, even be the most significant barrier.

Analytic marketers: It’s a small gene pool

Alas, there just aren't that many people who really understand – deep in their DNA – both marketing concepts and statistical validity. Tools can make it easy for people without this understanding to implement tests, but the ability to craft good test plans, and apply learning broadly is rare.  The parallel with Google Analytics is striking; it pioneered making web analytics very easy to use. But very, very few organizations who run Google Analytics use that data to drive decisions – there’s a talent gap, and it is only going to get worse.

Traffic volume: You can’t get reliable tests without lots of visitors

Sitecore 8 makes it very easy to implement a test, but most pages on most websites don't have enough traffic to make this practical – oftentimes, the test will fail to reach statistical validity within a reasonable time frame.

With the exception of very high traffic websites, testing everything is simply impractical. Consider the following example taken from a recent nonlinear engagement:

  • The site had a conversion rate of 2%
  • They anticipated that the tests they wanted to run might provide a 20% advantage over the current content

Given these numbers, a page had to have at least 2,300 page views per day for the test to reach statistical validity within 2 weeks. In practice, this mean that they could only test on the home page and one other page on the site.  

This example explains that, while Michael Seifret and Sitecore are taking steps in the right direction, there is a fundamental limit on testing everything, everywhere.

Stay tuned for more posts by our 6 MVPs attending Sitecore Symposium 2014!

comments powered by Disqus