10 Tips for Hosting a Successful A/B Testing Hackathon

There is no better way to find new testing ideas, generate excitement about testing, and push the boundaries of optimization than to host a testing hackathon.

An overhead view of TechCrunch Disrupt, one of the mega-hackathons.

You’ve probably heard of large scale hackathons like TechCrunch’s Disrupt NY or even seen company-wide hackathons promoted on blogs before. The gist of a hackathon is simple: put a lot of people in a big room together for two days with very few rules and see what they’re able to produce. The results are usually incredible.

At my company, Webs, we’ve taken that concept of collaboration, intense focus, and a shared purpose and applied it to testing and optimization. Instead of just developers, we invited our entire organization to participate in two full-day events that focus on brainstorming, building, and launching tests designed to make a transformative impact on our business.

Why are testing hackathons amazing? We’ve found that there is no faster way to strengthen testing culture, generate tons of diverse testing ideas, and build excitement around optimization in your organization than a testing hackathon.

Related Post: HackWeek at Optimizely in Photos

Still haven’t started testing yet? I recommend you check out Optimizely’s Roadmap to Building an Optimization Culture before you jump into hackathon planning.

Tip: Create posters to get people excited about the hackathon. Here’s an example from Optimizely’s recent HackWeek.

In our two testing hackathons, we’ve not only spread a passion for testing throughout Webs, we’ve also learned a lot about how to put on a successful one. I want to share those learnings with others out there looking to jumpstart their optimization!

1. Get management on board.

The first step of running a great testing hackathon starts long before the day of your event. Make sure to engage key decision makers at your group or company to get them on board with testing and the idea of a hackathon.

Need some selling points for a testing hackathon?  Try these:

• It creates an opportunity for different teams to work together.
• It is a positive boost to morale.
• It generates myriad new testing ideas.
• It has potential to drive positive impact to key metrics.

2. Make it a team event.

Hacking is a team sport! The more people involved — especially those who are not normally responsible for testing—the better. From our experience at Webs and Pagemodo, the optimal team size for a hackathon team is 3-4 people. At this size, each team is big enough to build some pretty sophisticated tests but nimble enough to change course if they need to.

Try to encourage teams with diverse skill sets (analyst, developer, marketer for example).  Teams with members from different functional areas are often more creative and successful than those that all look alike!

3. Introduce everyone to the testing tool before you start hacking.

If you’re like most organizations (including ours), not everyone is an expert on testing!  Schedule a lunch and learn during the week of the hackathon to share some testing basics like these: Optimizely Testing Basics. Also, use this time to add everyone to your Optimizely account and walk through a quick tutorial. This way people will feel more ready jump straight into hacking instead of getting up to speed on basics.

4. Get experiment idea juices flowing.

Have a testing backlog already? Great! Pick your top 10-20 test ideas and put them in a shared Google Doc. Include this list of potential test ideas with your team while promoting the hackathon to spark inspiration. Be sure to let them know that they can brainstorm their own, too.

Before the hack day, we’ve also found it helpful to have a few small brainstorm sessions with diverse groups of Hackathon attendees focused on the key pages of your site (Homepage, Signup, and Cart for example).

Ask questions like “What would a visitor want to do from this page?” or “How can we make it easier for a person to purchase something from this page?” You’ll be surprised by the creativity of your co-workers and all of the ideas you never thought of before!

We used this poster at our testing hackathon last year. Keep it interesting and fun!

5. Promote the event around the office.

When you’re promoting the testing hackathon, get creative! Hackathons are exciting and your promotion should be exciting as well. Make posters, talk about it at lunch, keep the event top of mind.

Also, make sure to let your coworkers know that the event is designed to be inclusive. All groups, marketing, support, developers, and even human resources, can make a big impact and their differing perspectives could lead to some great new testing ideas!

6. Set a flexible agenda.

Like any meeting or event, make sure everyone knows where they should be and when.  Share an agenda a week before the event and review it at the start of your event.

The agenda can be simple: ours usually include a start/end time, a break for lunch, demos, awards, and a post-awards happy hour. During hackathons at Webs, we’ve found it’s very difficult to get participants to break for lunch because they’re having such a great time hacking on test ideas!

7. Make sure someone is a floater.

As much as you’d probably love participating in the hackathon as a test builder, I’ve found that this is usually impossible as the organizer. You’ll spend a large portion of your time early in the event helping with setup and during the event, you’ll be busy helping others build tests, set goals, and launch tests successfully.

While the event is happening, try to talk with every team. Chances are, you can help them out with something! At our hackathons, I’ve filled this role and it’s been incredible having an influence on over 20 test launches in a single day.

Here’s a sample award for Most Creative Test we gave at our July 2014 testing hackathon. And, of course, every company needs a Testing Czar.

8. Give awards.

Make sure to recognize your co-workers for their achievements at the testing hackathon. Additionally, you can use these awards to help direct the groups where to focus. At our hackathons, we’ve given awards for:

• Most Sophisticated Test: What test has the most pages or interacts with the most distinct parts of your website?
• Most Creative Test: What test would you have never thought of yourself?
• Most Ambitious Test: What group set their testing sights the highest? Even if they didn’t end up achieving it, they deserve to be recognized!
• Best Overall Test: This is usually awarded 2-4 weeks after the Hackathon ends.  That way, you can measure which test had the biggest impact on your KPIs or metrics.

9. Have backup plans.

Like any event, no matter how well you plan, some things will inevitably go wrong. Make sure you have a backup plan ahead of time for these common issues:

• Can’t launch tests the day of the event due to site issues?  No problem! We can still build the tests at the hackathon, we’ll launch them in the future.
• Team members are sick? We can put you on a new team!
• Team wasn’t able to build their test completely in the hackathon? We can finish building the test later, but the hackathon still got that team much more familiar with testing.

10. Ask for feedback after the event.

Every event can get better. Make sure to send out a feedback survey the day after your event to see what people liked, what they didn’t like, and collect their suggestions for the next event. At Webs, we’ve learned many of the above tips from our follow-up surveys.

Get hacking!

After reading this top 10 list, you now have all the tools and best practices you need to run a successful testing hackathon. Hackathons have become a core part of our optimization strategy and our business here at Webs and I hope they can become part of yours!

Let us know how it goes and if you’ve learned anything else interesting with your hackathons. We can’t wait to see the pictures!