Get more wins: Why you should focus on experimentation now

In the wake of 2020 and the coronavirus pandemic, more companies are investing in experimentation and optimization than ever before, realizing that the ability to test and learn is vital to both pivoting as business needs change, but also to respond to adapting consumer behavior. With this new audience in mind, I’ll be writing a blog series called, “Getting More Wins,” outlining key elements of your experimentation program to focus on as you start and areas to invest in as you scale. In addition to discussing common tenets of building a program, I’ll be highlighting some of Optimizely’s best content on each topic, so this is truly a one-stop shop for intel on where to start and how to scale!

Why Focus on Experimentation Now?

There’s no better time to start building an experimentation program than right now – companies have witnessed dramatic changes in traffic, user behavior, and customer needs, and the brands that were able to pivot and react quickly were rewarded financially. We know consumers are behaving differently than they did a year ago, as each of us is personally behaving differently in our daily lives. In the past year at Optimizely, we’ve seen hundreds of experiments run in reaction to COVID across all client verticals and our suite of products. This makes total sense – no one had COVID-19 on their roadmap or a pandemic in their release cycle; Optimizely clients were able to react quickly, utilizing our platform to address changes in order fulfillment, increased interest in return and refund policies, and messaging around shipping and safety.

Even before the pandemic, research showed us that we should be investing in experimentation. The survey was completed in March 2019, a full year before the pandemic really took over most of our daily lives. We asked over 800 executives to quantify what extent experimentation impacted their bottom line – over 75% had increased revenues by 5% or more, with over 40% having increased revenues by over 10%. All directly attributed to experimentation!

Although independent research has repeatedly proved the value of experimentation, it still took a dramatic event to make experimentation a priority at many organizations. Now that we are all in agreement that an Experimentation Program is a non-negotiable part of your digital strategy, let’s cover tactical, tangible ways to grow from running a few tests a year to a true optimization program.

Over the next two blog posts, we’ll focus on three elements of your Experimentation Program to Focus on to Get More Wins – they center around people, metrics, and process:

• Two People Every Program Needs
• Two Sets of Metrics to Agree on and Measure
• Three Ways to Help Your Program Scale

The Two People That Every Program Needs

What makes a good program?

I grew up idolizing the 90’s Chicago Bulls (ala MJ & Phil Jackson) and came of age watching Duke’s legendary basketball program (the J.J. Redick and Coach K era), but we see great programs every day. One of the components most of them have in common is leadership – a Team Lead with charisma who builds cross-functional relationships and a Coach who can give direction and keep things on course. For your experimentation program, this is your Program Manager and your Executive Sponsor.

Why Do I Need a Program Manager?

Your program manager is your Team Captain: they see the whole court and run the plays day-to-day. They also serve additional functions:

• They are the biggest mobilizer for your program: driving internal communications & sharing insights and learnings with other teams.
• They own measuring, tracking, and reporting on your program’s metrics.
• They are advocating for resources and the Program’s direction, while also owning the Executive Sponsor relationship and incorporating their feedback.

We recommend you have a Program Manager that’s 100% committed to your program, but at the very least, you need to have one at least 50% committed to your program to truly scale to the degree to which most Optimizely customers aspire.

What is an Executive Sponsor? Why do I need one?

Similar to a coach, an executive sponsor is a leader inside an organization that takes responsibility and visibility for the success of any program or initiative. We’ve all seen scenarios where an organization wants to do something, but if it’s not an executive-level priority, it can often get stalled or sidelined due to lack of focus or funding. To encourage a culture of experimentation throughout the organization, particularly when it’s a new initiative, it will take an executive sponsor to drive adoption and change the cultural mindset.

There are a few key reasons why you need an Executive Sponsor:

• It’s significantly easier to build a Test and Learn Culture and encourage people to change their mindsets and workflows from the top down.
• They are the champion of your program: They can increase visibility for the wins, learnings, and incremental revenue driven by the Experimentation Program.
• They can unlock resources and advocate for engineering prioritization.
• They can help you build a coalition within the organization of allies and influencers.

Let me give you an example that illustrates the first bullet above (that a Test and Learn Culture can be easier from the top down): When I first started getting excited about experimentation at DIRECTV, I would push my working teams to iterate on our creative and have testing options baked into our proposals, but there was resistance – it felt like more work and many of our teammates just wanted to nail one direction for the Executive Creative review. However, a pattern started to emerge – in every weekly Executive Creative review, Clay, our VP and the Executive Sponsor for the Experimentation Program, would inevitably ask:

• “What are we testing this against?”
• “Are we going to test the order of these buckets?”
• “What about this price execution? Are we testing that?”

Suddenly, I wasn’t being an overachiever for wanting to bring test ideas to our reviews; having a point of view on what we were going to test and iterate upon became vital to having a successful creative review, which is basically your weekly goal when you’re at a digital agency. It created a huge mindset shift: people started brainstorming more and viewing success differently as well.

Who should be my Executive Sponsor?

For starters, an Exec Sponsor can sit in any part of the organization – both of mine were within Digital Marketing, but at Optimizely, our Exec Sponsors have traditionally been in our Product org. We also see SVPs of Analytics and SVPs of Growth serve as sponsors – it really depends on where your program sits in your organization and who your day-to-day users are.

A few additional questions to ask yourself:

• Who can quickly unblock some of the primary challenges (resources, prioritization) when starting your program?
• Who can immediately benefit from the data and learnings driven by experimentation?
• Who has a clearly defined KPI or metric that has high visibility within the organization?

At DIRECTV, we worked with two Executive Sponsors during my tenure – first, Clay Fisher, VP of Digital Marketing & Media, who really grew our program. Later, Seth Silverstein, Senior Director of Acquisition, stepped in and started helping us focus our efforts and also get the resources we needed. As the Senior Director of Acquisition, he owned a very important KPI for the business, gross new subscriber ads, that was reported up to our C-suite, stakeholders, and our stockholders on the quarterly earnings call. Similarly, he also owned our media spend and traffic-driving team, so the learnings from testing could be immediately applied to his SEM spend and landing page pages to improve his marketing spend.

You might be asking, okay, this makes sense, but what does this have to do with getting more wins? Getting more wins is really hard without a point person, someone building and iterating on the program’s processes, access to resources, and an initial direction for the program – all of which a Program Manager and an Executive Sponsor help you accomplish. You can aspire to have an amazing program, but you definitely need to also have the players on the court first.