Feature Flag, Rollout or A/B test?

So, when do you use one over the other?

This is a question I’ve heard from different developers, so I put together this high-level decision tree which is useful for determining when to set up a feature flag, plan a phased rollout, or run an experiment. If you find this helpful, this diagram is an example from my free e-book Ship Confidently with Progressive Delivery and Experimentation, which offers end-to-end guidance for deploying feature flags at scale. Now, let’s dive into the decision tree!

In general, it’s good to recognize that a/b tests are best when you have specific measurable metrics and a hypothesis for how your changes will affect those metrics.

Feature flags give you complete control over a release or launch date.

And rollouts mitigate risky changes by releasing code to small portions of traffic at once.

However, it’s important to note that certain feature flag and A/B testing platforms, like Optimizely, allow for all three capabilities to run off the same feature flag implementation. This means that you can decouple how you implement your feature from how you plan to release, experiment, and learn from it. See how this breaks down in the API call 'Decide':

[Editors note Nov-3-2022] isFeatureEnabled is now the 'Decide' function. See developer documentation for the latest code examples

You can also see how they all work together to support a strong Progressive Delivery practice across your entire product development team.

Let me know what you think!

How are you currently deciding when to use a feature flag, rollout, or A/B test? Message me in our Slack community or find me on Twitter at @asametrical.

This is part of a series of best practices to help your company successfully implement progressive delivery and experimentation to ship faster with confidence.

If you like this content, check out my free e-book: Ship Confidently with Progressive Delivery and Experimentation which offers more best practices from just getting started to scaling these technologies organization-wide.

And if you are looking for a platform to get started, check out Optimizely’s free offering.