At any company, there is a lot going on at any one time. People are very focused on their own efforts and what’s immediately in front of them. Experimentation, like any large company program, requires ongoing efforts to make the work and the impact visible. 

Increased visibility for an experimentation program can often come down to taking status out a digital place, bringing it into the physical space and frequently socializing the location of program updates.

Seek to have many channels where folks hear about experimentation in order to keep people thinking about it. This also increases your visibility to executives, who will enjoy seeing the excitement the organization has around this effort and the return on investment for experimentation.

Ideally, together these tactics demonstrate the entire lifecycle of experimentation. Where there are opportunities to iterate on the idea, what should be implemented as well as areas where we can be more thoughtful around metrics and overall experiment design moving forward.

This overall approach is powerful in that it doesn’t create new processes or meetings, but instead it seeks to leverage highly embedded processes and meetings that already exist. When you incorporate into existing workflows you also minimize friction that new processes often have.

Here are some tactical methods that I leverage for creating more visibility for my program.

1. Create an experimentation roadmap and socialize it across the company

Use a digital source of truth to track a large backlog of experiment ideas. This will track what’s been deployed quarter over quarter and what’s getting prioritized. This is also helpful for distributed teams that aren’t all located in a single place where a wall of work can be seen. All this information can be captured in an experimentation roadmap.

Optimizely’s own Program Management product is helpful for effectively running our internal experimentation program. It keeps all the information about our program in a single place and is open to the entire company, so anyone can submit ideas and provide feedback.

We also require an idea to be complete in advance of presenting at Experiment Review, our weekly meeting reserved exclusively for discussing product experimentation at Optimizely (more on this later), which enforces completion of all sections around Goals, Hypothesis and Metrics. By requiring this, the conversations can be more focused and fruitful, so having a system for idea backlog management is helpful in multiple ways.

I’ve also seen teams use a Google template that outlines “Experiment Requirements” that team members must complete and submit in advance of experiment review meetings. From there, they can use a spreadsheet to track their experimentation roadmap.

Documentation [here]

Another method I’ve seen teams use is leveraging JIRA for tracking experiments. This is most common on product and engineering teams as this allows experimentation to fit more seamlessly into their current processes.

For me, tracking experiments that are started, especially quarter over quarter, is one of the most important utilities for an experimentation roadmap.

2. Incorporate experimentation into current workflows

Any time a product manager sits down to write out the “why” behind a new feature, they have to complete a Product Requirements Document (PRD) and present it to the entire Product team. As part of that process, they now also have to include what exploratory experiments they have run to help validate the problem area:

If your team has processes around Product or Technical design documents, add experimentation to that template to encourage hypothesis thinking and experimentation.

3. Physical walls represent the state of the program

Although I monitor our experimentation program closely, not everyone at the company does. Having a physical wall that displays a summary of all experiments running and planned makes it easy for the whole team to see status toward our shared goals and makes it easy for me to have discussions about the state of the experimentation world and blockers.

At Optimizely, we have a large wall of work that represents all of our Engineering work in flight, which is lovingly maintained by my boss, Keith. In addition to having all engineering features on note cards moving through our build and release system, we also leverage this space for tracking updates to large company goals like feature utilization, number of monthly active users and, for me, experiments that are planned or currently running.

This is especially useful as all engineering managers and product managers were already meeting weekly to discuss work in flight at the board, so having this incorporated on the wall of work folds experimentation into that conversation seamlessly.

I leverage space on the far side of our overall engineer wall of work. This board is a win-win in that is also helps make the overall program more visible and can prompt hallway conversations as folks are walking by.

I’ve taken it one step further and created cards, similar to those used for features in flight on our wall of work, to display the status of experiments as they are going through our delivery system. This helps to ensure Product Managers are mentioning them as they are discussing everything that is in flight during our big weekly standup. Having these cards also better visualizes where in the dreaming → designing → developing → delivering process these experiments sit.

These cards are also helpful for me to be able to see the system as a whole and if there are enough potential experiments across the system to hit our quarterly targets.

Adding the cards to our current system is powerful in that it doesn’t create new processes or meetings. Instead it leverages highly embedded processes and meetings that already exist. Find ways to incorporate into existing workflows to eliminate the friction that new process often creates.

4. Encourage experimenters to share their results and insights

It’s important to ask those closest to the experiments to share their results. This helps diversify who the company is hearing about experimentation and allows those working on those experiments to share the best insights directly.

Encourage members of the team to share experiment findings or new data insights at all-hands meetings. This helps keep the program top of mind and gets folks inspired to contribute. I lovingly call this tactic “Creating a Culture of FOMO (Fear of Missing Out).”

Slack is a great tool for keeping the team informed about the program. Below, you can see Evan sharing his excitement about his statistically significant winner and all the great feedback via emojis he’s getting from the team. Slack is great lightweight way to share ongoing updates about the program to keep it top of mind for the team.

I also have all experimenters send company-wide emails when an experiment starts and share the results when it is stopped.

5. Hold a weekly meeting to discuss best practices

While I aim to not create additional process or meetings outside of what already exists, having a space where your core team and experimenters can get together to uplevel everyone’s experimentation practice is powerful. This meeting, which we call Experiment Review at Optimizely, also creates a specific space to discuss experimentation best practices every week, allowing for upleveling of specific experiments as well as space to discuss how the program can iterate.

Use this meeting to ensure that every experiment starts with a strong hypothesis, actionable metrics, and good experiment design. At Optimizely, we use experiment review as a final step for the experimentation team to give the go-ahead to start an experiment.

I also love having experimenters come back to share results once they have run their experiments. This helps show the entire lifecycle of the experiment, where there are opportunities to iterate on the idea, what should be implemented and areas where we can be more thoughtful around metrics and overall experiment design moving forward. This is also a great way for folks to share their experiments while also continuing to uplevel around results interpretation and metrics to leverage.

Below, I’ve included an example of an agenda for experiment review, which I share via a Slack room weekly.

6. Use large company events as an opportunity to hype experimentation

Leverage opportunities to present experimentation work in large company meetings to let your team know that experiments are moving the needle for the business. Updates can also drive engagement by getting new team members excited to experiment. Again, creating a “Culture of FOMO.”

All-hands meetings are a great way to get in front of your executive team and share the power of experimentation.

A rule of thumb that I live by is that unless people see you walking up to give a presentation and everyone is thinking, “This person again? I already know everything they are about to say,” you are not speaking about your program enough.

You can also share program updates with your team broadly on large Slack channels to increase your reach. Not everyone can attend meetings in person, so leveraging asynchronous methods that have large audiences is important. Below, as an end of quarter recap, I shared all the great new content my core experimentation team had generated with our sales team and noted how it would be most interesting to them, which is sharing with their prospects to help close deals.

I hope this gave you some new ideas for how to better socialize your experimentation program! Any of these tactics of engaging with the organization will get people more involved and aware of what is going on.

Are there any tactics you find especially interesting? Any that I missed that you use yourself? Comment below or tweet me @bexcitement.