With the recent surge in popularity of the Lean Startup philosophy developed by Eric Ries, many people have become familiar with terms such as MVP (minimum viable product, the least-effort deliverable that can be used to validate assumptions with real customers).
Split or A/B testing is another Lean Startup concept and one that we’ve been using at Go Daddy for as long as I’ve been around (9 years this month!). A split test is an experiment on a website in which some visitors are shown a modified version of a page or pages while others continue to see the “old” version. Some goal, such as increased clicks or conversions, is measured and compared between the test and control groups to determine if the new version achieves the desired goal.
Split tests are a great way to validate changes, but they’re also a challenge. Ideally, we want to run a large number of tests and quickly learn from each one. But, changing something on a website – especially a highly dynamic eCommerce site like GoDaddy.com - requires effort and care. This work can be minimized by using yet another Lean Startup concept called continuous deployment (I’ll save that topic for another day). Ultimately, to perform the volume of tests we want, we need to go from continuous deployment to no deployment.
A project I’ve been working on recently led to an “Aha!” moment: there are products out there that do exactly this for websites. These products fall under the generic category of “multivariate testing software.” There’s even a website that does comparisons and reviews of these tools at http://www.whichmvt.com (I’m not endorsing the reliability of the ratings on this site – do your homework).
To gain insight from a split test, you have to measure the impact it has on your goal. The testing systems we reviewed all contained their own tracking mechanisms and reports. Most can also be integrated with an analytics service like Google Analytics. The result is a complete solution that puts the power of split and multivariate testing into the hands of the marketing and product teams, enabling a lean startup practice in your organization.