How to Run an Experimentation Hypothesis Workshop with Your Design Team

When starting your program, you may not have built out your backlog of experimentation ideas just yet. Or you may need to fill your backlog with ideas from new parts of the organization. Maybe you’ve been really focused on iterating on successes and it’s time to take a step back and refill the tank with ideas.

Both with our customers and internally at Optimizely, our Strategy team runs hypothesis workshops to fill an experiment backlog. One of the core principles of a strong experimentation program is that “no idea is a bad idea” or, in the space of this workshop, “All ideas are safe in the workshop.” So when we hold these workshops the goal is clear: let’s think of any and all solutions to the customer problems we’ve identified. These problems are the most important part of getting through the Problem, Solution, Result framework.

Disclaimer: there certainly are bad experiment ideas. Those ideas without proper problem definition. Those fueled by misinformation or no qualitative or quantitative data. When we highlight “no idea is a bad idea” we mean that you should open up the program so everyone has the ability to submit a good experiment idea

You should bring those teammates who are closest to and have a deep understanding of the customer problems you are looking to solve from experimentation. Especially since those running an experimentation program aren’t always customer facing, bringing in people in customer-facing or in customer-centric roles like Customer Success Managers or Product Designers can yield a fresh experimentation perspective since they actively hear direct feedback from your customers. For a Design Team, this should be the folks who have lived closest to your user research practices and inform the work of the product and engineering organizations.

You’ll need to define the roles within the workshop. There are only two!

1. Facilitator – Typically the program manager, the Design Team lead on experimentation, or in some cases an outside consultant. The facilitator will ensure the workshop goes along the agenda and the outcomes are met.
2. Ideators – All other team attendees who will be tasked with coming up with solutions (ideas) to fill the backlog.

As the facilitator, you will need to ensure your “Ideator” teammates come prepared with defined customer problems. Teach your team that if they mention your product, service, or solution in the customer problem statement, then it’s probably not a true customer problem. You must first define the problem before even getting to a solution. And a great problem has data validation written directly in.

Here are some example problem statements:

• A poor problem statement: Customers want to be able to pay bills using their smartphone camera.
• A good problem statement: All mobile banking customers want the convenience of easily paying bills on their mobile device but are required to enter payee details for every single transaction.
• A great problem statement: All mobile banking customers want the convenience of easily paying bills on their mobile device but are required to enter payee details for every single transaction. We see a 45% drop off on the first form field entry and an average time on page that is 70% site average.

These should be backed in at least a single data point. Your Design Team will have a great sense of what customer problems exist from their usability testing, customer interviews, and other customer feedback loops. But provide them with other data points that may not be utilized as frequently.  These can come from your analytics systems, heatmapping tools or retention mapping. At Optimizely we provided our own Design Team with our current performance on our north star goals and access to our Amplitude reporting along with those goals.

You can send this template to your team to prepare them. And can also share with them an experiment design canvas template so their mind is already seeing the output you are looking to get to in the workshop.

For the facilitator, you just need to come with your own view on the customer problems (do your own homework!) and some sticky notes. More on how you’ll use those next.

So, your “Ideators” have done all their pre-work and the day of the ideation workshop is here! You can use this framework as the facilitator for your workshop. Give each team member a set of sticky notes as they’ll be writing out both their problem statements and ideated solutions for them.

Here’s how to breakdown the use of an hour workshop (you could spend a whole hour on problems or solutions, so feel free to break these out into their own individual workshops or extending time of a single workshop if it helps!):

1. [10 minutes] Team members write out problem statements with data point(s)
2. [10 minutes] Review problem statements and group by theme then prioritize
3. [15 minutes] Team members write out solutions to prioritized problem statements
4. [15 minutes] Review solutions and group by theme then prioritize
5. [10 minutes] Group discussion on results (metrics) to determine success on the solution
   1. Keep this focused to just the goals that are necessary to calling a winner and loser, nothing more! These should include macro-level goals (final conversion, revenue, etc.), but also those macro conversions tied closer

For both the problem statements and the solutions, take the sticky notes from each team member and read them aloud for everyone to hear and discuss. Ask the submitting team member to clarify those that aren’t super clear or are interesting enough to have them expand on the problem for the group. Then post them on the board/wall and start to group them by theme.

Themes should be similar problem statements that are aligned to improving the same north star goal. For example, these problem statements all fit in a similar theme:

• Customer feedback from our voice of customer tool shows that mobile users find the time spent paying online typically takes longer than emailing or calling
• Form field completion rate for mobile users paying their bills has declined quarter-over-quarter; we typically return many errors during this process
• In usability testing, we found that mobile users do not notice the ‘Save Your Billing Info’ feature for easier payment in the future

These all carry the same theme and could be summarized in a singular customer problem supported by the above data-first problem statements:

Customers who use their mobile phones for banking find paying bills tedious and the process of entering the details for each bill cumbersome and time-consuming.

You should wrap up the workshop with a good set (shoot for about 5-10 of completely different and sane solutions) of new hypotheses. You’ll most likely have a longer list of ideas and solutions that you can keep in your pocket to revisit later. But ensure you get through at least 5 really well thought out hypotheses. Now you’ll need to get the hypotheses into the next step of the methodology: prioritization.

Though you may have only discussed on a smaller set of solutions due to time constraints, make sure the facilitator captures all written down by the group (“no idea is a bad idea”). You can use this two-tabbed spreadsheet template to capture all the Sticky Notes before you walk out of the room.

The primary next step for the Design Team is to take those that were discussed in the room and put them where you prioritize your backlog. You can move your ideas into Program Management as Optimizely does and ask each member to then score them and comment at the rest of the team to do the same. These should then make it for discussion in your next weekly experiment review meeting!

Here’s a good way to email the Design Team with those calls-to-action.

As any meeting with many stakeholders, these meetings never go quite as smoothly as you originally envisioned. Here are a few things to keep in mind as you enter the room as the Facilitator:

• If there is a problem statement or solution you feel is interesting or needs clarification from the group, read it out loud. Ask the person who wrote the problem or solution to add a bit more color for the group.
• Don’t talk solutions during the problem statement time. Problems first! Solutions come later.
• Though the primary goal is to walk out with a good set of new hypotheses, the secondary goal is to help the team understand how to walk through the problems, solution, result framework. If it means focusing on a smaller set of problems and solutions to reinforce this, that’s more than okay.
• If you’re short for time without talking results in detail that’s okay. After the prioritization process, you’ll have time to really define a strong test plan (link?) for those hypotheses that have risen to the top.
• Play it loose and fun! This may be the first time you’ve all gotten in the room to talk about optimization like this. Make sure everyone has a chance to participate. This is how you build advocates for the program and get a diverse set of ideas to experiment on. You never know what may work.

It’s a good experimentation practice to run a hypothesis workshop once a month to keep the backlog deep and fresh. Another good practice is to start opening up the invite list over time– include other functions that are working on solving the same problems. We’d suggest you run these independently with each functional team first so they get the rhythm (and to limit the number of stakeholders who are inexperienced with the process and ensure a successful outcome). 

We also suggest running a hypothesis workshop with your executive team! This may only happen once a year, but is a great way to get executive buy-in and some fantastic ideas. A good time is typically at the start of your year as you define north star goals and OKRs. What does the executive team think would move the needle? You’ll probably need to provide a little bit more assistance on what data can help define a problem statement, so we’d encourage you to make it a longer workshop and commit at least 30 minutes to talk through the problems.

————————-

What is unique to the way your experimentation program runs hypothesis workshops? How many ideas do you shoot to walk away with at the end? Let us know!