How to Use Data to Choose Your Next A/B Test

Every aspect of life is a prioritization decision. Did you work late last night and skip the gym? I did. Did you recently cancel a vacation because you had a family emergency? I did.

Apparently I care about work and family. If I didn’t, I would have taken different actions. Every action you take reflects your priorities in a world with limited resources, and our single most limited resource is time. How we spend our time says everything about what we care about.

Time is also the most limited resource in web optimization. In order to execute an optimization campaign, you need two things: website visitors and good variations to show them. Both of these actually boil down to time. The more visitors you need, the longer you need to keep a test running. And, the more complex your variations, the more design and development time needed to build them.

When you’re choosing what tests to run next, you need to prioritize. In this post, you’ll learn how how to scientifically prioritize your testing efforts to ensure the tests you’re running will have the maximum impact on your conversion rates.

Before Getting Started

Before getting started, you should choose what to optimize for and determine that the exercise is worth the time investment.

Agree on a Success Metric

John Egan, a growth engineer at Pinterest, knows a thing or two about how to optimize an on-site experience. He recently wrote a blog post titled Why You Should Be Using ROI To Run Your Growth Team. In it, he makes a very straightforward point: prioritize your efforts based on possible returns for the business.

It sounds almost silly when you put it like that. The whole goal of on-site optimization is aligning visitor behavior with business goals. You want visitors to to click buttons, fill out forms, and take a million other actions that are all associated with the success of your business. Of course tests should be prioritized based on ROI!

With that decision made, you can start to look for ways to estimate ROI for a given test. More on that later.

Decide to Invest the Time

Artisan commerce giant Etsy uses data extensively when prioritizing new features. John Perkins, a product analyst at Etsy, frequently runs analyses to help product managers assess the potential impact of proposed features. “We calculate the potential impact for every new feature before building it. Why build something if it won’t make a real impact on our business?”

Pinterest does it. Etsy does it. The problem is, most of us don’t. Or at least, not scientifically. It’s tempting to choose what to test without a rigorous evaluation process.

It’s easy to understand why this is: it takes time and energy to rigorously prioritize tests by evaluating potential impact. Designing and implementing a truly great test takes time and a few extra steps. The good news is that effective prioritization with data actually saves time and achieves better results in the long run, even if it takes more investment up front.

Estimate the Impact of a Test

Once you’ve settled on the fundamentals, it’s time to start planning. How exactly will you calculate the potential ROI of a given test? When calculating potential impact, Etsy measures the total population that would be impacted by the feature, and then the potential performance improvement if the new feature is successful. Multiplying these together get them an estimate of what a given feature could be worth. At the highest level, this approach is quite simple:

Here’s a spreadsheet template that takes this broad concept and builds upon it. if you’d like to use this spreadsheet in your prioritization efforts, just go to File > Make a Copy and you can get started.

Where do I get the data?

Your web analytics tool. Most marketers use Google Analytics, so I’ll focus on that. Google Analytics doesn’t have a standard report built just for this purpose; it requires a bit of work to get there. Good news is, I’ve created a shortcut for you. This Google Analytics custom report contains a list of your website’s pages, the number of pageviews, and the count of conversions. From that, you can easily calculate your current conversion rate (number of conversions divided by the total pageviews.)

While you’re looking at the data in this report, sort by most pageviews. Grab the top 10-20 rows. Your primary targets for optimization will likely be living in your highest-traffic pages. Once you know your top pages, pull your goal value data for each of these pages. If you don’t have goal values set up, here’s a great post on how to get started.

Now it’s time to come up with good hypotheses to test on these pages. That’s way beyond the scope of this post, but here’s a post from Optimizely on how to build a data-driven hypothesis.

What about the Target Conversion Rate?

In the Google Analytics report, you’ll notice that there’s one column in the above spreadsheet that isn’t included: Target Conversion Rate. That certainly isn’t a metric you can find in Google Analytics, so what is it?

If you want to determine the value of any given test, you ultimately need to make an educated guess on the results you could potentially achieve against a certain baseline. Without a conversion rate improvement, you won’t be able to calculate test value. This is a difficult prediction to make. A/B test results are full of surprises, so it’s impossible to know with certainty what the result of a given test will be. But with a rigorous thought process, you can make some reasonable estimates.

To make a well-informed estimate of a conversion rate improvement for an experiment, here are some guidelines:

• Draw parallels from other similar pages on your site. If you have another page that has a similar function but performs better, you might be able to replicate this conversion rate performance.
• Draw parallels from other tests you’ve run. Have you measured the impact of similar copy changes or design changes elsewhere? You might be able to get similar conversion improvements here.
• Look up benchmarks for page performance for businesses like yours. This technique is tricky, because every site is different, but if you don’t have any other relevant experiences to draw from it can be a decent place to start.

It’s important to remember that your conversion rate improvement is an estimate, and doesn’t need to be a perfect prediction of how your test will perform. This approach doesn’t rely on you making the perfect guess; it’s likely that the same high-priority experiments will rise to the top of a queue of tests even if your estimates turn out to be off. As you continue to use this approach moving forwards, you’ll start to develop and hone instincts about the size of improvement a given test will have.

Now that you have the data…

The strength of this approach is that once you’ve collected the data, you have a very cut and dry mechanism for prioritizing what tests to work on first. Just sort the whole spreadsheet on the value column from largest to smallest. Voila! A work queue. Start at the top and work downwards.

Potential Refinements

This approach is a simple one, and leaves plenty of room for improvement once you get up and running. This is actually a strength. It’s better to start with something simple rather than to get bogged down in complexity from beginning.

Once you’re ready to refine your process, here are some suggestions.

1. Incorporate time estimates

Remember ROI? ROI is benefit divided by cost, and we never estimated the costs. In this model, the costs are implementation time. Some tests, like changing headline copy or font sizes, are trivial to implement. Some tests require more effort from your designers and developers.

If you take the prior model and add an estimate for the number of hours it would take to implement a given test and the rate-per-hour you’d like to use, you can come up with total implementation costs for a test. Then, take the potential benefit and divide by the implementation cost to arrive at test ROI.

2. Consider multivariate

Sometimes you’ll find that your highest traffic page—typically your homepage—has the highest potential value tests on it. And when you rank order your tests by priority, you may find that many of the top ones are all on the homepage. The problem is that it’s not typically feasible to execute multiple A/B tests on a single page concurrently.

Multivariate testing allows you to understand the effect that multiple variables will have on a single conversion goal. Multivariate testing is a godsend for optimizing high-value pages. At RJMetrics, we’re proud Optimizely users. In fact, our very own Stephanie Liu won the 2014 Testing Hero of the Year award! Last year, when we kicked our site optimization activities into high gear, we prioritized running multivariate tests on our homepage because it is, quantitatively, a very high-priority area for us to be testing.

The payoff has been huge. As of this writing, we’re about to implement a multivariate test on our homepage; one test is on the copy and one is on the button color.

Go Forth and Prioritize

That’s it! I hope I’ve shared why it’s so important to prioritize on-site optimization activities and some tools to get started. There are certainly many more refinements to be made as you become an expert in this field. In fact, Etsy product analysts will actually spend multiple hours, days, and even sometimes more than a week attempting to assess the impact of a given test.

The value of a good estimate only increases as the test that you’re running gets harder and harder to build. If you have a team of five engineers working on a test for two weeks, you’d better be certain that’s the best use of their time.

But don’t let perfect be the enemy of good. What’s most important is that you’re being rigorous about what to test and in what order. Now let’s get to work.

If you’re looking for more ways to use data to grow your business, don’t miss RJMetrics’ ebook, 22 Ecommerce Growth Hacks.