Product delivery: How to use experimentation to deliver features without risks

A product delivery process focuses on delivering features fast. 

As product managers, if you want your features to stick, test them first with product experimentation. It ensures product delivery without risks or bugs.   

It doesn’t matter how fast you ship a feature. If they don’t align with the organization's goals, you’re likely to face failure.  

That’s where a test-and-learn approach comes into the picture. It challenges the usual ‘Success depends on praying’ approach and quantifies customer impact early and often.

Using feature flags and  a/b tests for product validation and ensuring everyone’s headed in the right direction.    

The roadmap process you have:  

Design -> Build -> Launch -> Pray   

The roadmap process you need:  

Design -> Build -> Experiment-> Customer feedback-> Iterate -> Customer satisfaction  

Deploy quicker. Feel safer.   

To deliver a great product, combine feature flagging and an experimentation framework to gradually roll out big changes or even small changes. 

Feature flag development gives product development teams full control over rolling out on rolling back in production for a safe testing process independent of code deployment.    

A big customer pain point is that any experiment brings down the application which affects the user experience. Feature flagging solves that issue and works like an assurance plan around preventing that from happening. If something does break down, you can roll the feature back quickly and minimize damage.     

It enables new features to be hidden from end-users and tested internally. You can release features to specific groups or percentages of users. Acting like an "off switch" that quickly addresses bugs, it saves time and allows developers to move faster and more confidently with high-quality products.    

Here are 7 reasons to use feature flags:   

• Risk management   
• Canary releases   
• Simplicity   
• Server-side A/B testing   
• Feature gating   
• Continuous deployment   
• Feaster release cycles     

Feature rollouts   

Once you are ready to deploy a new feature, a feature rollout comes into play. It helps you control who sees the new feature and enables you to grant access to a specific feature to a certain set of users.   

You can randomly assign to a percentage of users, target a specific audience, whitelist individual users/accounts, or do a combination of all these. With rollouts, you can ramp up access to any feature or experience slowly, to monitor key engineering metrics like performance or bugs, or Key Performance Indicators (KPIs) of the user experience impact.      

Feature test   

Feature tests allow product teams to iterate further on product design and features post-launch. Once a parameterized feature has been deployed to production, any business or technical user can Optimizely's feature tests to experiment on different configurations of the feature without another deployment.     

This enables users to run more experiments and iterate on a feature to find the best version, not simply the one that was the MVP at the launch.      

Progressive delivery   

Going beyond agile and continuous delivery, here is a way for product owners to validate quality and performance in production.    

Why opt for progressive delivery?   

• More effective collaboration among team members and stakeholders 
• Reduce risks in software-relates   
• Target specific audiences through multiple stages of release      

When releasing a feature, you can first release it to them followed by a beta release. If that's successful, conduct staged rollouts before finally releasing the feature to everyone. Your product roadmap timelines should be divided into rollout sprints.        

How to measure user engagement?   

If you are considering adding a new feature to your product, consider running different types of tests during the project management phase to assess its usability. A painted-door test is a great option. It is a quick and easy way to get valuable feedback from your users and make sure that you are building the right thing. You can gauge if your users would engage with a particular feature.      

The test works by creating a minimal user experience that mimics a full feature set. For example, you could create a button that says "Coming Soon" and leads to a page that explains the new feature. You can then measure the number of users who click on the button to see how much interest there is in the feature. This allows you to validate if this is something a customer truly wants in your new product, or if you need to re-evaluate the idea.     

Behind the painted door, you can also collect additional qualitative data. It can help you understand what users hope to achieve by using the product or feature.    

If you do want to do a painted door test, here are the best tips to keep in mind:   

• Get direction- small traffic allocation   
• Learn quickly- stats accelerator   
• Extra analysis- use additional a/b testing tools   
• Be nice- clever copy helps   
• More insights- focus on user needs and continuous improvements   
• Lastly- just build and iterate      

To ensure everything goes smoothly, you need product experimentation to complement progressive delivery.    

Time for product experimentation   

It puts customers at the front by allowing for quick iteration at every stage. You can:   

• Start with rapid prototyping in the ideas phase   
• Shift to risk-free rollouts in the build and release phase   
• Add optimization and personalization in the launch and measurement phase.   

The best part about this methodology? You get real-time test results and analysis. It helps ensure that your company is making the best decisions possible to continually offer the best customer experience.    

Test, learn, and adapt with data   

Reaching statistical significance is important as it fosters data-driven decisions about what features are more effective in helping you reach your goals during a product lifecycle. Better testing methods mean more clicks, views, profitability, and sales for your company.  

Without statistical significance, you might be prone to random decision-making without evidence support. Peeking at your results many times increases the chance that you find results that look significant when they actually are not. On the other hand, sequential tests become more confident with each other once your tests reach significance.   

Building a test and learn culture   

Here are 3 steps to prioritize a test and learn culture at your organization and improve product delivery.     

1. Set clear milestones    

In your product delivery process, identify your goals from managing product delivery and gathering good trusted insights and data. Consider how they map back to the phases in the product development process. It'll help you understand where should you put your resources for maximum output.      

Here’s what this testing strategy could look like:   

• Test and iterate on ideas to improve product quality   
• Quickly identify the highest-value product opportunities   
• Reserve engineering time for the most important development   
• Validate code quality and performance   
• Prevent negative experiences for the target audience 
• Introduce the right features to the right audiences, in the most impactful ways   
• Use data to understand your customers and deliver a successful product

2. Evaluate   

This step is about figuring out which areas of the tech stack need work to achieve those goals.    

So, it's better if you:   

• Understand where in your technology ecosystem feature flags and experiments provide better results for your business.   
• Know how to plug gaps in your data pipeline.   
• Can find ways to enhance existing internal platforms.    

3. Create innovation model   

Staying ahead of your competitors is about understanding customer needs and being data-driven about every decision you make in the process. Technology and data automate innovation and can help you get the test & learn approach right.      

So, make sure to build the right kind of foundation. Iterate and provide value. Question every step and test. Building the right questions into product development will help you develop features and functionality that'll balance customer wants and business value in terms of ROI.      

Remember...   

Feature flags and experimentation go together. They provide a way for platforms to serve the needs of an entire product development or DevOps team. This continuous user testing helps organizations put their customers at the front and center. After all, customer experience and business value are all that matter.