0:00
/ 0:00
Table of contents
Scaling your experimentation program is like planning a mountain climb—you need more than just quick wins; you need a strategy. While A/B testing is widespread, there’s still limited guidance on how to grow these programs effectively.
Based on data from over 127,000 experiments across 1,100 companies, we’ve distilled six essential tips to help you shift from quick results to lasting impact. These insights will guide you in making smarter, data-driven decisions to elevate your experimentation efforts.
Tip 1: Set the right metrics
Here’s the truth about experimentation: Not all experimentation metrics are created equal. Over 90% of experiments target 5 common metrics:
- CTA Clicks
- Revenue
- Checkout
- Registration
- Add-to-cart
However, the data shows that 3 of those top 5 metrics have relatively low expected impact.
Loading Chart...
Source: Evolution of Experimentation Report
Could you accidentally be ignoring metrics that can make a difference? For example, the search rate has a high impact even though it is hardly prioritized.
For more impact:
- Find decision points that lead visitors to the buying moment.
- Choose metrics that affect each decision point.
- Deliver high-impact with each metric.
Impact -> More uplifts -> Higher sales.
Tip 2: Increase testing velocity
The whole point of experimentation is you're not guessing anymore. However, most tests only focus on simple changes. For example, button color & copy. Measuring testing velocity is a great way to get momentum. The median company runs 34 experiments per year. The top 3% of companies run over 500. To be in the top 10%, you need to be running 200 experiments annually.
Loading Chart...
Source: Evolution of Experimentation Report
However, when scaling, the focus needs to shift from velocity to bigger changes and impact. Here’s why we recommend moving beyond running simple experiments:
- No long-term learning
- No hypothesis apart from "what if..."
- No major uplift or impact on business goals
Loading Chart...
Source: Evolution of Experimentation Report
They:
- Make larger code changes with more effect on the user experience (>99.9% significance)
- Test a higher number of variations simultaneously (>99.9% significance)
Experimentation acts like the operating model of your business.
Tip 3: Run more complex experiments
Loading Chart...
Source: Evolution of Experimentation Report
- Over half of all variations tested on checkout test one single change type
- Variations with more changes offer greater expected impact on checkout
So, how do you set up your program for success?
- Have more developer resources with a diverse portfolio of iterative changes (pricing, discounts, checkout flow, data collection, etc.)
- Document every change and improve how users interact with your website/app and their behavior directly.
- Choose experimentation metrics that measure operational efficiency, quality, and program adoption - depending on your company's needs.
See how William Hill invested 3 years in growing its experimentation program by shifting to an agile mindset and data-driven decision-making.
Tip 4: Embrace personalization
One size fit all is no longer a viable digital marketing approach. You can't just push the same website recommendations to a broad audience.
Ask yourself: Do you even resonate?
Still, most digital businesses avoid personalization due to resource constraints, uncertainty about customer preferences, and the complexity of implementing tailored experiences.
To find out more about the impact of personalization, Optimizely even surveyed 100 marketing leaders and 1,000 consumers, about today’s personalization practices.
Check out the personalized to personal guide.
The case of 41% more impact
Personalization can help you elevate customer satisfaction, expedite purchasing decisions, and boost web conversions.
Looking at the lessons learned from 127K experiments, half of all experiments today use a personalization strategy. It generates a 41% higher impact compared to general experiences.
Loading Chart...
Source: Evolution of Experimentation Report
When personalizing, keep in mind:
- WHAT: The change you want to make in the default digital experience
- WHO: The specific user or group you want to deliver it to
- WHY: If the change meets the original objective/goal
- WHERE: The platform you’ll use to create a personalized experience
Personalization examples:
- Send targeted offers to shoppers for their favorite products based on browsing behavior.
- Offer travel promotions for different locations based on the current weather or season.
- Show video content to viewers based on where they live and what they search for.
It's all about creating customer journeys that provide a comprehensive view from the customers' perspective.
Learn more about optimizing experiences as Juliana Jackson, Data Strategy & Digital Experience Optimisation Practice Lead @ Media Monks, shares invaluable insights on transcending siloes and mastering stakeholder collaboration.
0:00
/ 0:00
Tip 5: Protect your developer resources
You will never have enough resources to build everything you want to do. So, how do you decide what to develop?
Experimentation can help you filter, evaluate, prioritize, and de-risk.
Tests per developer
The highest expected impact occurs at 1-10 annual tests per engineer. Move beyond 30 and the expected impact drops by 87%.
Loading Chart...
Loading Chart...
Source: Evolution of Experimentation Report
Volume at the cost of quality can harm performance and the expected impact of your experiments.
That is why you need to carefully plan how to use your developer resources. It requires:
- Setting clear hypotheses and sharing them with the experimentation team
- Carefully planning resources
- Providing autonomy for decision-making
Quality -> Velocity -> Impact (and conversions)
Reduce time to value with AI
It's hard to know what can this new AI thing do in the world of experimentation. If you don't have enough developer resources, you can still do a lot with an experimentation platform that uses AI capabilities well.
You can:
- Use the AI code generation capabilities to enable your team to run experiments.
- Have automated guidance around which parts of your site to test and what to measure.
- Set segment-based recommendations for your users where they're more likely to buy.
Learn more about AI capabilities and what they can do for your experimentation program in this video.
0:00
/ 0:00
Tip 6: Build a strong culture of experimentation
Running tests and an experimentation program are great, but it’s critical to develop and foster a culture of experimentation across the business to truly supercharge your program.
So, what stops companies from doing so?
- Other teams using out-of-date practices shift the focus to smaller improvement opportunities.
- Leadership overestimating their ability to influence the future creates friction.
- Lack of enough quality data means most tests don't reach statistical significance.
When scaling, your experimentation program needs to balance exploitation and exploration. Allow your teams to take the right risk when the opportunity occurs.
Learn more: How to build a culture of experimentation
Stakeholder management and structure is key to success
Having a close relationship with the changing priorities of the wider business is essential for the prioritization of your tests and the growth of your team. Avoid being siloed.
Ensure other teams have learned the fundamentals of what makes a good experiment. It will help you decide who can run an experiment, review results, and ultimately determine if a winner has been implemented.
To learn more, check out this leadership guide to experimentation.
Better insights. Sharper analytics.
"The data must tell us something"
"We're data-driven."
Loading Chart...
Source: Evolution of Experimentation Report
- Teams with Analytics outperform teams without by 32% per test
- Teams that added heatmapping were an additional 16% more successful
- Given that not all companies with analytics use it effectively, this suggests that analytics usage is a major improvement opportunity for more companies
But most data you see and use is layered with assumptions and subjective interpretations.
Nothing wrong with it, but if you operate without any awareness of the user's needs, you'll act even more subjective than you set out to be.
Web analytics isn't just an interrogation of data. It's about having the ability to think critically through assisted use of data.
In short, experimentation helps you build thought processes that are more important than the data itself.
- Breaking down business problems into parts and steps
- Designing analyses and experiments
- Recognizing and avoiding cognitive bias
Check out this video to understand Optimizely's world of experimentation alongside tactics on how we fine-tune our own strategies for success.
0:00
/ 0:00
Keep learning
If you implement just 1-2 of the learnings outlined in this article, you’ll be well on your way to scaling and driving even more value from your experimentation program.
Ready to take a deeper dive into the world of experimentation?
Here’s what we recommend:
- Full report: Evolution of Experimentation
- How to build a winning experimentation program
- Product experimentation guide