Using Feature Flags to Test in Production

Frank Hu – Senior Engineer, Optimizely – Tests his code in production

One of my favorite developers used to tell me that he does robust testing on his code, but only in production. This may give some QA engineers a heart attack, but I’m here to share a little secret: at Optimizely, we test in production all the time. If you aren’t, you may be short-changing your customers.

In most organizations, automation and manual testing occurs in the staging environment. Engineering organizations do this because testing in production can leak issues that expose vulnerabilities to the public, or create a bad user experience that damages the reputation of the organization.

But engineering organizations that don’t test in production aren’t actually checking whether their features work in the actual environment in which their customers are using them.

The synthetic data that is used in staging environments will never behave 100% like real data and bugs often arise from this discrepancy.

So how should QA teams test in production without exposing ourselves to these shortcomings? Our secret sauce is that we use our Feature Management solution to deploy our code and then use our Feature Flag Audience Conditions as a permission toggle to add in our testing teams.

The full process looks something like this:

1. Developers deploy code behind a feature flag with the feature turned off. The new features are now deployed into production, but are hidden from our customers.

2. QA creates Audience Groups and add in an Audience Condition with our Testing Group

3. This Testing Group consists of Customer Service Managers, Solutions Architects, and Business Users. These users have the most connection to how our customers use our product, or how to support our customers who use our product. They also have data in their accounts and use cases that most developers and QA engineers do not.

4. QA team holds a bug bash with this Testing Group and allows everyone to bash against the new features to look for bugs.

Since the new feature is governed by a feature flag, if no major bugs are found and we want to deploy to the public, we can roll it out to all users. In Optimizely, you can do this in a Feature dashboard with two clicks and save.

If major bugs are found, we simply turn the feature off or remove the Testing Group Audience Condition. This allows the users in the Testing Group to be put back in the pre-release code path without having to actually do a code rollback.

At the end of the day, we should test our code against the environment that’s most similar to the one our customer actually uses, without exposing them to any actual bugs. With feature flags, not only can we always test our code, but we can always test in production!