Optimizely Debuts New Report Revealing Increased Rates of Experimentation

NEW YORK, November 27, 2023 - Optimizely, the leading digital experience platform (DXP) provider, today unveiled a new research report, “The Evolution of Experimentation” that delivers an analysis of the over 127,000 true experiments conducted on Optimizely Web Experimentation and Optimizely Feature Experimentation between 2018 and 2023​. With this report, Optimizely hopes to advance the practice of experimentation around the world by sharing valuable insights and guidance for digital practitioners.​ 

The report reveals that since 2018, Optimizely has observed significant growth in the number of experiments its customers conducted, an uplift in the volume of companies experimenting, and a 227% increase in the usage of feature experimentation as a percentage of all experiments. While data shows broader adaptation of experimentation in recent years, substantial differences remain in the quality of experimentation occurring across the landscape.  

“This data confirms what many at Optimizely already know — the best in the game embrace experimentation to deliver more exceptional digital experiences to their users,” said Eric Lang, Senior Consultant of Strategy & Value. “There is no doubt that every test uncovers valuable information on customer behavior, no matter how small an experiment may seem. Those who experiment gain key insights into behavior in order to deliver exceptional digital experiences and develop long-lasting, personal relationships with their customers.” 

The results of the report underscore the value of a robust experimentation operation for any digital brand. The data shows initial ideas implemented by digital marketers are very rarely successful. According to the report, 88% of the ideas implemented by digital teams are not winners​, meaning they do not result in a positive significant change in user behavior. However, even if a test is not a winner, it provides valuable insights into customer behavior and allows teams to eliminate guesswork and pivot quickly.  

“Experimentation allows digital businesses to achieve the highest level of customer-centricity by checking all of their ideas with the only audience that matters – their customers,” said Pete Gray, SVP of Customer Growth Strategy at Optimizely. “The emotional cost of experimentation is to learn how wrong we can be. The prize is the ability to never make a mistake again and to unlock real value by increasing the velocity of improvement.”

Additional key takeaways from Optimizely’s new study include: 

Contrary to industry advice that good experiments should be minute in scope, the highest-performing experiments have very different characteristics.​  

• Experiments that test multiple treatments are at least 1.5x more successful than simple A/B tests​  
• Experiments that make significant changes to the user experience are 25% more successful​ 
• Experiments that are personalized create 41% higher impact 

To be in the top 10% of experiment velocity, companies need to run over 200 tests annually​. 

• The median company runs 3 experiments per month​ 
• Reaching the Top 10% of velocity requires scaling to over 16 tests per month​ 
• Only 3% of companies have joined the elusive 500 tests per year club 

High and low performers exist in all industries, and all companies have the potential to improve their performance​. The average experiment win rates by sector​ are:  

• Food and Beverage: 17% 
• Automotive: 16.2% 
• Hospitality and Travel: 15.7% 
• Media and Entertainment: 15.5% 
• Financial Services: 15% 
• Real Estate: 14.4%
• Healthcare and Medical: 13.3%
• Sporting and Recreation: 13.2%
• Retail and Distribution: 13.1%
• Telecommunications: 12% 

To read the full report and learn how Optimizely can help your team run experiments, generate insights, and continuously optimize experiences across the board, visit https://www.optimizely.com/the-evolution-of-experimentation/