How to build a culture of experimentation

Running experiments at scale is a superpower.

Get it right, and you’ll grab attention and see the impact on business goals that directly benefit the company, including sales growth, more conversions, cost savings, etc.

Fail, and all the hard work you put into your customer experiences will go unnoticed (and unanswered).

But what exactly is a culture of experimentation?

It's an organizational mindset that embraces continuous testing, learning, and iteration to drive innovation and improvement. In this article, we'll explore how to build this culture and why it's crucial for your business success.

Why do you need to build a culture of experimentation?

Initiatives like testing and optimization don’t happen in a vacuum – they’re surrounded by other variables and insights. Here, it’s about testing your entire digital experience across different business units, devices, and channels.

While you focus on turning visitors into paying customers, you need to think about the following:

• If you want to A/B test across different devices, do you have the right tech and key metrics to enable it?
• If you’re A/B testing across the enterprise, do you have a governance structure to measure results?
• Do you have the right stats and results you can rely on?

Before diving into the how-to, let's establish the foundational elements of a successful experimentation program.

Many organizations have an internal debate about the most important metrics, but it is important to focus on the right top-line goals. For example, leading indicators are important to test, but often poorly understood and under-tested.

Teams should focus on the following experimentation metrics and resources:

Key metrics

• Company-wide and business-unit Key Performance Indicators (KPIs)
• Benchmark performance metrics. The list includes conversion rate, Revenue Per Visitor (RPV), Lifetime Value (LTV), Average Order Value (AOV), Return On Investment (ROI)
• Leading indicator metrics. The list includes test velocity, efficiency, quality, and more.
• Program budget

Essential resources

• Program Manager
• Team leads (technical, design, analytics)

Critical actions

• Benchmark KPIs and baseline modeling
• Conduct goal tree exercises
• Align with stakeholders on optimization program goals

Learn more about scaling and running successful programs in our lessons learned from 127k experiments.

So, how do you use these adaptability metrics to build a culture of experimentation?

The key is having trusted measuring pillars you can rely on to improve your company culture.

6 tips for building a culture of experimentation

Here are six techniques you can use for brainstorming sessions when building a test and learn framework:

1. Test the entire digital experience

   Use innovative ideas to create curiosity that motivates visitors to take action.

   • Optimize every touchpoint in the customer journey and remove roadblocks that stop them from buying 
   • Use deep personalization methodology to build credibility and sell new products
   • Connect the dots between different stages of the buying process

2. Optimize experiences on every device

   A good digital experience can adapt and impact user experience across devices.

   • Adapt user experiences across platforms (web, mobile apps, single-page apps, TV apps, IoT, voice apps, etc.)
   • Use segmentation to create experiences based on user needs
   • Implement a system that allows easy changes across platforms without requiring a developer

3. Make better decisions faster

   Ensure the validity of your test results with data.

   • Use a Stats Engine to increase testing velocity
   • Calculate sample size to improve the accuracy of results
   • Make your tests data-informed to detect a difference at any moment

4. Research + data-driven marketing > Opinions

   The goal is to improve conversion rate by optimizing each metric.

   • Use data to drive decision-making and eliminate subjectivity
   • Make it easy for anyone to interpret the results
   • Continue iterating on the need for data-driven approaches

5. Introduce experimentation early

   Build product experiments quicker. Introduce experimentation early in the product development lifecycle. 

   • Prove assumptions before building, prioritize what matters and deliver features without risks 
   • Use open APIs to create your ideal tech stack
   • Integrate easily with existing systems to act on customer data

6. Best practices work if you make them work...

   Combine a shared vision with collaborative tools

   • Implement experimentation and learning in your work environment
   • Gather, track, and report on experiment results across the enterprise
   • Assess the quality of your tests and their impact on business success

Experimentation beats speculation

Even if you're a small business with only a few stakeholders, these pillars will serve as your guide to continuous improvement and innovation.

Key takeaways:

• Focus on the right metrics and resources
• Test across your entire digital experience and all devices
• Make data-driven decisions quickly
• Integrate your experimentation tools with existing systems
• Continuously learn and apply best practices

And if you still don’t have an experimentation culture or a startup, we can help you start from scratch and achieve positive results.

Check out this 2024 Big Book of Experimentation. It's packed with 40+ industry-wide use cases and case studies showing how real businesses leveraged experimentation to build a positive experimentation culture.