How to Run Losing Split Tests and Still Win

By showing different variations of a product or feature to segments of your audience, you can quickly learn what works and what doesn’t. There’s a reason that every major tech company such as Google, Apple, Uber, etc. Use split testing as a part of their design and research process. Done correctly, you can learn about your customers rapidly and increase revenue. When done wrong, split testing can become an expensive waste of time.

The Wrong Way to Split Test

Here’s what happens a lot when someone reads an article on ConversionXL and decide to start split testing:

• They install a split testing tool like Optimizely or VWO.
• They stop throwing spaghetti at the wall and seeing what sticks.
• Instead, they throw two pieces of spaghetti at the wall and seeing which one sticks better.

When you test this way, you’re missing out on a majority of the value split testing provides. You need to approach split testing with the right mindset to get good results.

The variant on the left with additional parmesan converting 300% better.

Testing is Learning First, Revenue Second

The more you understand your customers, the better you can serve them. To learn the most you can, you need to approach testing with a scientific mindset. Instead of saying “Let’s try something,” or “Let’s pit Tom’s idea against Sarah’s and see who wins,” Start with a hypothesis. Say “If we make this change, I think this effect will happen because of this reason.”  When you approach testing this way, every test will be a win regardless of income:

• If you lose, then you’ve disproven a notion, and can you this information to inform future tests.
• If you win, then you’ve found a way to increase conversions, and you have a notion of why that is. You can use that information to inform future tests.

Before you run a test, ask yourself these questions:

• What change do I want to make?
• What metrics will it move?
• How much will it move this metric?
• Why will this work?

Don’t Let Logic Stand In the Way of Thinking Outside The Box

Since we are testing, any reasonable hypothesis warrants consideration. If there is a justification, go for it. If you have a reason puppy GIFs would increase conversions with your audience, go for it. Ideas should come from anywhere in the company, not just the design team, marketing team, management. Employees working closer to customers and the product will often have insight other departments don’t. When coming up with test ideas, use data and research to back up your points, but keep an open mind.

Treat Every Test Like a 3rd Person

Don’t take test results personally.  Don’t use the phrase “my test” or “my idea.” Talk about your test in the third person. You’re a scientist, remember? Treat your thoughts with a healthy, cold, logical distance. You’ll need to because…

You are Going to Lose. A Lot.

Most of the time, split tests lose more than they win. You’ll have ideas that you think are surefire winners that will bomb horribly. Think of it like baseball. The best hitter in the 2016 season missed 65.2% of his at-bats. Losses are temporary. Wins are evergreen.

Celebrate Those Wins! ?

Often the biggest challenge isn’t what to test or how to test, but getting buy-in that testing is the way to go.  Split testing can be uncomfortable for some organizations; you want to help alleviate that awkwardness by highlighting the benefits. Even though you aren’t taking things personally, it’s still nice to let yourself have a moment to enjoy when a test goes well.

No Matter What, Share Your Findings and Show Your Work

The more people that benefit from your testing, the better,  When you get results, win or lose, share them with the rest of your team. Marketing, support, design, and management can all benefit from the data you can gather via testing. Keep sharing an open and company-wide process.