The 6 Types of Feature Flags You’ll Meet at Optimizely: Bug Fix

Tl;dr

🚩 How do we use Bug Fixes at Optimizely? To safely test fixes for more complex issues.
🔒 Access Level? Everyone.
😬 Risk Level? Low.
👩‍💻 Tests? Test it like a new feature (unit/integration/end to end/manual).
⏰ Lifetime? Short. As soon as the fix is working, remove it.

Most bug fixes need to get out immediately through a hot fix or incremental release (you broke something and the customer is not happy, times ticking!). However, what if the issue is a bit more complex and you have a few different solutions that you could put in place. Which one will be the right solution? Which one could potentially make things worse? In situations like these, placing a bug fix behind a feature flag can allow you to test in production to a limited amount of customers (contain the blast radius), check to see if your fix is performant (check against  production traffic), and finally, if it actually improves the situation (feature test different solutions against each other). If you got it right, great (roll it out)! If not, you at least contained the blast radius (read on more below), or roll it back.

How do we use Bug Fix Feature Flags at Optimizely?

A feature flag to bug fix is when developers ship a fix to a bug in production using a feature flag. The benefit of leveraging a feature flag for bug fixing is it allows the engineering team to experiment with multiple fix solutions and see which fix is the ideal solution by deploying a few different solutions behind a feature flag and use production traffic to see which solution is the best fix.

Using a feature flag to bug fix is the most rare flag we use here at Optimizely. We do not use feature flags for minor bug fixes or emergency bug fixes (hotfixes/patches) as we find this creates unnecessary overhead, they are only used in the following ways:

• We find a more complex bug in production. The team wants to test the fix in production, but still wants the capability of disabling the fix in case it causes the situation to degrade further.
• The team has a few different solutions and wants to test which one is most performant. They can deploy all the solutions behind a feature flag and use production traffic to see which solution is the best fix.
• We want to test our fix in production against an internal QA Account, and if it works, then roll out to production.

Who is allowed to make changes to Bug Fix Feature Flags?

At Optimizely, feature flags for bug fixes are treated the same way as feature rollouts and are handled identically. All engineers have access to these feature flags since multiple development teams may be contributing to the flag. We use our Change History feature to audit who made these changes and updates to the linked Jira ticket to give us clarity over what is happening with this bug fix.

What is the risk level for using a Bug Fix Feature Flag?

Feature flags for bug fixes have low risk (we’re not introducing anything new, only fixing something currently not working). We also manage the risk using the following methods:

• Deploy first to a canary account with only internal users.
• Rollout the bug fix to a smaller number of customers. Monitor the behavior and if the fix is not performant, roll back.

By utilizing audience targeting and slow rollout of the bug fix, we can confidently ship out our solution and ensure it doesn’t make the situation worse.

How does Optimizely test its deployment of Bug Fix Feature Flags?

Optimizely tests its feature flags for bug fixes identical to its feature flags that are rolling out new features. Shipping code for new features is treated exactly as shipping code to fix a broken feature (roll out to targeted audience, test, and once confident, roll out to completion). 

Before we put in a bug fix behind a feature flag, we normally do an audit over why our original unit test, component test, end to end test, or manual sanity checking did not catch the original bug. We ask questions such as: “Did we forget a certain scenario?”, “Did we not understand the feature as our customers used it?”, “Did our test not work correctly?” and ensure that we have the proper changes to our tests to address these questions. 

When does Optimizely remove its Bug Fix Feature Flags?

Optimizely Kanban Board showing our Feature Flags and what state they are in.

As soon as we have confirmed that the bug fix has actually resolved the initial bug, we mark the bug “ready to be removed” in the linked Jira field. Some teams will then have the engineer on support rotation remove the feature flag, or some teams let it get picked up to be removed during Feature Flag Removal Day.

This is the part of a series that will dive more into the different feature flag types we have here at Optimizely, as well as the engineering teams that implement them. See you next time as we dive in deeper on Operational feature flag type.

If you’re looking to get started with or scale feature flags, check out our free solution: Optimizely Rollouts! 

Are you using bug fix feature flags at your company? I’d love to hear about your experience! Twitter, LinkedIn