To Move Faster, Experiment More

It always takes longer than you expect, even when you take into account Hofstadter’s Law.

So it’s no surprise that every product development team wants to be more “agile.” To the naive CTO, “agile” is just a synonym for “fast.” It’s a magic keyword that speeds up the treadmill, telling a team to run faster, launch quicker, and do more with less. And if that’s all agile means in your organization, then you’ll be disappointed when you hit a wall of missed expectations, burnt out employees, and stagnant metrics.

That’s because proper agility isn’t the same thing as speed. It’s the prerequisite for speed, not the end result. Agility is the confidence you need to move quickly. Most engineering teams aren’t slow because they’re lazy, because the work is too hard, or even because they’re understaffed. They’re slow because they’re afraid — and usually for good reason.

The fear I’m talking about here is the fear of failure. If your product has any level of adoption, then there is oh so much that can go wrong. Like a house of cards, an application takes years to build and a second to knock over. There are a million ways to break your site, tank your metrics, and annoy your users, and precious few ways to be sure you’re not making that kind of mistake. So the more our product grows, the more afraid we become. And ironically, this fear of failure ends up being precisely the thing that prevents success. We get worried about going the wrong way, so we move slowly and cautiously. We get nervous about bold decisions, so we wait for consensus. And we start to fear change, so we cling to the status quo.

There’s something very primal about this fear. It’s as if we’re afraid of the dark. Think about the last time you had to find the bathroom with the light off, or walk outside without a flashlight.

When we’re trying to move in the dark, we walk tentatively. We hold our hands out in front of us and shuffle slowly, afraid of tripping on unseen obstacles or bumping into painful barriers. We take careful steps and feel around for clues, hoping we don’t get lost. We cling close to the group and follow the loudest voices because we don’t see a clear path. And no matter how hard we try to stay together, we inevitably drift off in different directions.

Now imagine your boss, standing in front of all their employees stumbling around in the dark, shouting “GO FASTER! WHY AREN’T YOU RUNNING?” It can only makes things worse. They will drift faster, collide faster, trip faster, and fall faster — but they won’t actually achieve anything faster because they can’t see the path or the destination. If this is what it feels like to develop software in your organization, the problem isn’t with your people, it’s with the tools they’re trying to use to move around in the dark. Slowness is a product of fear, and fear is a product of uncertainty.

Now imagine a flashlight. What if every employee could flick a switch and suddenly see everything — where they are, where they’re going, and what the path looks like from here to there. Then suddenly they wouldn’t have to shuffle and stumble. They could start to run, confident that each step would get them closer to their destination.

This is the lesson we see over and over in agile development. When engineers have automated tests, they can refactor anything without worrying about what might break. When teams adopt frameworks like Scrum, they can adjust their approach in the middle of a project to adapt to customer demands. When launches are gated behind feature flags, anyone can roll out a bold new idea, secure in the knowledge it can be rolled back at the first sign of trouble.

What each of these practices points to is the essence of agile product development: a culture of experimentation. This kind of culture empowers your team with the confidence to make bold changes by constantly validating your ideas with data. Experimentation is the key to agility because it makes you fearless. It’s the flashlight that illuminates the whole product development process, giving your team the clarity it needs to build the best possible product for your users.

What does this mean concretely? There are many ways to experiment, from simple A/B testing to elaborate multivariate approaches. It’s a cultural mindset more than a specific tool, but generally it requires a few key practices.

The first step is establishing metrics. You can’t develop products effectively if you don’t know what you’re trying to achieve and how well you’re tracking towards those goals. What does success look like for your product? To answer that question, you’ll have to work backwards from your company’s business model — which might be based on subscriptions, advertising revenue, in-app purchases, or some other activity that drives success for the business. For any given feature, you should find concrete metrics that capture its impact toward those goals, like increasing retention, driving actions like reading and sharing, or reducing churn from high-value customers. Whatever the metric is, you have to measure it reliably and then hold your team accountable to improving it.

Once you have metrics, you can start using hypothesis thinking. Instead of a “waterfall” roadmap made of dramatic launches, develop theories about how your users behave and where your product can go. For example, rather than declaring “we have to redesign our signup flow,” you might ask, “why do only 22% of new users sign-up?”. You could then consult your metrics, analyze the data, and form a hypothesis, like, “if we had a simpler sign-up flow with fewer steps up-front, our users would sign up at a higher rate because they’d get value from the product sooner.”

As Satya Nadella of Microsoft puts it:

If you are going to have a risk-taking culture, you can’t really look at every failure as a failure, you’ve got to be able to look at the failure as a learning opportunity….Instead of saying “I have an idea,” what if you said “I have a new hypothesis, let’s go test it, see if it’s valid, ask how quickly can we validate it.” And if it’s not valid, move on to the next one.

There’s no harm in claiming failure, if the hypothesis doesn’t work. To me being able to come up with the new ways of doing things, new ways of framing what is a failure and what is a success, how does one achieve success — it’s through a series of failures, a series of hypothesis testing. That’s in some sense the real pursuit.

Armed with a clear hypothesis, you can start validating ideas through gradual rollouts and A/B testing. For every UI you design or API you build, the question to ask is: “did we accomplish what we set out to do?” To even pose this question, you must first have defined your goals, measured them with metrics, defined the problem, identified a solution, and gated that solution behind a feature flag. Only then can you validate it by bringing it in contact with your users, as early and as frequently as possible. You can validate a new feature through gradual rollouts, monitoring your metrics as you go, or you can measure it directly by running an A/B test where half your users get the new feature and the other half don’t.

And finally, you can use experimentation to iterate through variations of an idea. The term “A/B testing” is really a misnomer, because the most effective experiments have more than just two variations and proceed through many cycles of improvement. Instead of a linear product roadmap, with each feature either launched or killed, imagine a forking tree, with each feature tested and tweaked through many variations. One reason to try many possibilities is that you’re more likely to find a better outcome. Optimizely’s data shows that experiments with a single treatment have just a 14% chance of significantly improving metrics on average, but tests with four treatments double this chance to nearly 28%.

What these numbers should illustrate is that most experiments fail, and that’s the point. By testing many hypotheses, and many variants of each approach, you steer your team towards the biggest impact for the business. You also learn valuable lessons along the way, driving future hypotheses for more ambitious experiments over time. A/B testing is often misunderstood as purely a trick for driving clicks on forms. But experimentation is broader and deeper than that. The best product teams deploy it not just as a tool for increasing conversions, but as a cultural practice that unlocks agility by giving engineers and product managers the confidence to take smart risks.

So if you want your team to move faster, challenge them to experiment more.

This is the first in a series of articles on how to build better products through experimentation. Our goal is to demystify the process of A/B testing and feature rollouts, so please share your questions and feedback in the comments. And if you’re ready to take your product experimentation to the next level, try Optimizely Full Stack.