Failing brilliantly: The secret weapon for experimentation success

Remember when running an experiment was as simple as changing a button color and watching the conversions roll in?

Yeah, those days are gone, just like flip phones and dial-up internet. 

Right now, things are tough. Budgets are tight. Job security feels like a distant memory. HIPPOs are keen on you to deliver statistically significant results and show value, fast. 

But here's the thing about experimentation - every time we fail, we grow. 

I still remember my high school haircut. It was a massive failure for me :) 

Source: Tenor

Don't be afraid to embrace failure and learn from every test you've run.

In this blog, we'll explore how to turn failed experiments into valuable learning opportunities.

When optimizing, there are no failures— only chances to learn and grow. Discover how to extract insights from every test, successful or not, and build a culture of continuous improvement in your experimentation program.

The reality of experimentation: Making failure cool again 

Failure didn't stop Vincent Van Gogh from painting masterpieces, Albert Einstein from revolutionizing physics, or Michael Jordan from becoming a basketball legend.

Yet, it has paralyzed countless experimentation teams, preventing them from reaching their full potential.

Why? In the high-stakes world of business, where the success of metrics drives the budget, failed experiments are often seen as costly mistakes rather than learning opportunities. This mindset stifles innovation and holds teams back from achieving breakthroughs. 

But can you guess what is the average win rate across experimentation programs? 

It is Barely 12%. 

Common failures include statistically significant but negative outcomes, inconclusive results, and flawed test designs. 

And it hurts. It feels like: 

• You wasted time and resources 
• You're going back to square one 
• You weren't able to prove someone wrong 

However, failure isn’t a negative thing, at least in experimentation. You can reframe failure as a positive. 

Why you ask? 

Each failed test allowed you to validate a feature or an experience that you’d otherwise spend a bunch of time and money building out. 

Plus, you likely learned more/something new about your user's preferences. 

Still don’t believe in the power of failure? 

We’ve got 5 examples from real practitioners to prove you can learn interesting things from every experiment you run.

Hall of fame: 4 Learnings from failed experiments 

1. Low-interaction content matters

Felix Kirmse (Digital Marketing Manager, Optimizely): Felix wanted to reduce low-interaction content on the start page to increase form submissions. 

• Result: Form submissions decreased further (uh oh) 
• Takeaway: Even low-interaction content can contribute to user trust and engagement, which are essential for encouraging form submissions.

2. Real book cover vs. Abstract version 

Joe Geoghan (Senior Visual Brand Design Specialist, Optimizely): Joe wanted to test the real cover vs. an abstract version of the big book of experimentation in the email body. Assuming the real cover would win, it was the cover used in most of the emails. 

Version 1

Version 2

• Result: The abstract version still ended up winning 
• Takeaway: The small version of the cover was too small to show the important information -- the illustration and text were both illegible. The abstract version was concise and showed you exactly what you were getting into. In design, clarity matters. 

3. Images aren’t always better than links 

Jeffrey de Wijs (Digital Marketing Manager, Optimizely): Jeff, the one who sends the Optimizely newsletter to your inboxes, last week tested if using images instead of bullet points, will increase clicks in our newsletter. 

• Result: Images and links did not perform better than links alone 
• Takeaway: Most team members liked the image variant more, but still there was no real difference in clicks between them. So, don’t trust assumptions. Go out and test it out yourself. 

4. Customer perspective: Predicting the next best action 

Virgin Media (Optimizely Customer): Charlotte Golding and her team at Virgin Media wanted to predict the Next Best Action (NBA) so they could design personalized experiences for their customers. They assumed customer would only have specific requests like improving the network in their area or upgrading their existing broadband, etc. 

• Result: The team found that the same customer would come in with different requests each day. One day, they were looking for customer care and the next day, they just wanted to upgrade. This wasn't initially factored in the NBA but after the experiment, the team had to optimize their model to better understand on which next best action to show to a customer. 
• Takeaway: Customers can come to your website about a different thing every day. Don't just put them in a single personalized experience and expect the same results. Optimize the model regularly. 

What does this show you about reframing the concept of failure in experimentation? 

To start, it is about fostering a risk-positive culture. 

Here's how to do that:

1. Give time and space to run tests at scale: Balance speed with thoroughness to ensure meaningful results. 
2. Start from the top: When executives champion experimentation, it becomes part of company culture. 
3. Aim for both moonshots and incremental gains: Combine quick wins with bold experiments for maximum learning and impact. 
4. Encourage team members to share ideas: Create a system for collecting and evaluating ideas from all levels of the organization. 
5. Celebrate "productive failures": Experiment Collaboration transforms experimentation into an integral part of your company culture. By standardizing workflows, sharing plans and results openly, and enabling anyone to participate, you create an environment where testing becomes second nature.

At a famous brewery, the CTO's F*UP award recognizes all well-designed experiments, regardless of the outcome.

This approach encourages the team to innovate boldly, try new things, and openly share all learnings - not just successes. It turns every test into a valuable opportunity, fostering a culture of continuous improvement and breakthrough discoveries. 

Cultivating a continuous culture of experimentation 

Overall, impact is more important than win rate. However, to learn from failed tests, you need to set appropriate risk thresholds and embrace a culture of experimentation. Here’s how to do so: 

• Redefine metrics: Move beyond win rates. Track insights gained, hypotheses validated/ invalidated, and long-term impact on key metrics. 
• Make it easy to run tests: Optimizely Web Experimentation and Feature Experimentation allow teams to run tests without heavy reliance on technical resources. 
• Implement a learning repository: Document all test results, insights, and follow-up actions. 
• Run review sessions: Hold monthly meetings where teams present their most impactful tests - both wins and losses. 
• Set clear risk thresholds: Define acceptable levels of risk for different types of experiments. For example, allow higher risk for tests on low-traffic pages or segments. 

Learning from every test you run

In experimentation (as in life), failure can be a powerful tool for growth. Starts with the individual - you can take practical steps to change your perspective and take from failures. But also, leadership needs to establish the right conditions to bring about the learnings that come from failure and not see it as a misstep / negative. 

Remember: 

• Impact trumps win rate. Focus on insights and long-term improvements. 
• Make experimentation accessible across your organization. 
• Document and share learnings to build collective knowledge. 
• Set clear risk thresholds to empower calculated risk-taking.