How Obama’s Campaign Team Sourced A/B Tests from User Feedback
When thinking about A/B testing, the pivotal question is what to test. Ideally you should focus your testing efforts on the page elements that are most likely to have an impact.
A great source of discovery for high impact elements can come from focusing on the user experience. Usability studies, user experience research, heuristic evaluations, and surveys can all yield insights into where users have difficulty (and success) using your site. By focusing on fixing the issues, and doubling down on the successes, you’re guaranteed to run tests that benefit your audience. In return, you’ll achieve a big win for your users, and a big win for your business.
This technique was used to great effect during Obama’s campaign to increase donations. Kyle Rush, Obama for America’s Deputy Director of Frontend Web Development, wrote a wonderful article about all the different testing ideas his team implemented. All the low hanging optimization fruit had been picked on their donation page, but they knew from usability best practices that the form was too long and losing potential donations. When it comes to donations, federal law requires specific information, which meant the team couldn’t just throw out unnecessary form fields. Instead, they set about making the form appear shorter by breaking it into pieces. And rather than just making the order of fields arbitrary, they went the extra step by analyzing validation errors of each field in order to put the easiest ones first. The final order went
1. Donation amount
2. Personal information
3. Billing information
4. Occupation/employer.
They had two simultaneous hypotheses working in concert: first, breaking the form up into smaller pieces makes it less intimidating to complete; and second, by having the easier fields first they made it easier to start filling out the form. Additionally, validation occurred each step of the way, which made finding and fixing errors easier since visitors only had a few fields to check at a time, rather than all 16. Finally, asking for the donation amount first more closely matches the user’s state of mind. Once they’ve made the decision to donate they’re ready to enter an amount, not their personal information.
Original long form on the left; optimized form broken into four parts on the right. The optimized form yielded a 5% conversion increase.
The team then proved their hypotheses were correct through testing. By breaking the form into four smaller parts the donation conversion rate increased by more than 5%. The team successfully identified a usability issue, implemented a solution, and proved that their solution worked through testing.
By analyzing the user experience, the Obama team was able to focus their testing resources on a change that was highly likely to prove successful. They didn’t waste any time on tests that lacked a solid hypothesis. By taking insights from the user experience you can generate high impact testing ideas that are highly beneficial to users and your bottom line.
Rationally Irrational: Academic Findings You’ll Actually Use is a series that provides practical testing ideas based on the study of how people make decisions (formally known as behavioral economics).
