The Secret to Experimentation Program Success:Sharing Broadly

You did it. You’ve gotten buy in and you’re off to the races and running great experiments! Like a tree falling in the forest, however, if people don’t know about it – how will they remain invested in your success and the success of the experiments?

I’m the Program Manager for Experimentation at Optimizely and I’ve seen the benefits of having both pre-defined templates and a set cadence for sharing experimentation updates. This keeps the visibility for the program high at the same time it simplifies the process for those launching experiments to send these updates themselves.

Having this process in place is especially helpful when you have a more decentralized experimentation program, be that a Center of Excellence or a Testing Council. These models provide experimenters with process, best practices and the all-important visibility for their work.

Common Types of Program Structures

Source: Building a Culture of Experimentation at HP

There are certainly other ways to share updates such as speaking at large company gatherings or simply setting a meeting to discuss experimentation. But as people digest information in varying ways, having multiple asynchronous channels to communicate about your program ensures the greatest number of people will be both apprised and invested in your success. Below are templates your team can leverage to effectively share your experimentation efforts, read on!

1. Experiment Kicked Off

Informing your company every time an experiment is launched is standard best practice. This heightens and maintains the visibility of the program. The extended team knows experiments are being launched consistently and that experimentation is still an important, active program. Additionally, seeing others are contributing to experimentation and getting company-wide visibility can encourage additional employees to contribute via the “Culture of FOMO,” and clarifies who to discuss experimentation with. Finally, informing of customer-facing UI changes cuts down on surprises when demo-ing or at customer on-site visits. At Optimizely, we love being able to share with our customers that we are “drinking our own champagne” and experimenting throughout our product and on our marketing landing pages,. It is equally it’s vital to let our internal teams know as well.

These emails are, by design, lightweight. Below is an example of one I’ve sent at Optimizely as well as the template we leverage internally. As you can see in the title, we mark these emails as “[visible-change][experiment]” to clearly call them out in the subject line.

I recommend sending these in advance of the experiment launching and here at Optimizely we send it company-wide.

Optimizely Example

Template you can use [here].

2. Sharing Results

Ok. Experiment launched. Email sent. Now, your experiment has results! This is a great opportunity to follow-up on your experiment start email and close the loop with the company about the impact experiments have had. This also is the time you can show the experiment is tracking back to its metrics as well as larger company KPIs. Which, in turn, is showing the incremental ROI of the experimentation program.

Just like the experiment start emails, these emails can be lightweight. I’ve included one I’ve sent at Optimizely as well as the template. From a process perspective , I have our experimenters respond to the previous experiment start email, include details on the results and next steps.

I recommend sending these when you are ready to pause the experiment based on statistically significant results and/or learnings that you are ready to take action upon and share with the team. Again we send these company-wide.

Optimizely Example

Template you can use [here].

3. Quarterly Recap

Sending this recap is the most important of the three email types and templates. It provides an the opportunity to demonstrate the overall ROI of your experimentation program. Showing the aggregate value of the program, tying it back to top line business goals and showing the additional value of the program. These can be things like training, onboarding or ideation sessions which show the broader value of the program to the whole company and, importantly, executives. I can personally attest that my CEO responded to the last quarterly recap update email to congratulate my team on our great progress!

Assuming you’ve had your team send the ongoing Experiment Start and Experiment Results emails, these recap emails are fairly simple to create.

I recommend sending these at a consistent cadence. At Optimizely, we do quarterly goal setting of Objectives & Key Results (OKRs), so I send an update of the program against our OKR goals to align with that cadence. Aligning to processes you already have in place is the simplest way to think about the timing of this update.

Template you can use [here].

As with any process, getting the muscle memory for you and your team to send these ongoing updates might take time, but stick with it! Visibility for your program, informing your company of ongoing changes and demonstrating ROI is extremely valuable.

I hope this gave you new ideas and templates to more easily socialize your experimentation program. Are there any templates you utilize internally? Any other tactics you use to socialize your program asynchronously? Comment below or tweet me @bexcitement.