Build vs. Buy: Choosing the Right Experimentation Solution

Engineers and product managers are empowered to A/B test each new product feature in a controlled environment to determine impact to key metrics like engagement, product usage and revenue before launching the feature to everyone.

These companies have all built complex internal software platforms to support their experimentation-driven approach to product development. If you’re looking to adopt the same practice of validating digital product decisions with data, you’ll likely be evaluating how to implement an experimentation platform for your product and engineering team.

Depending on your company’s unique requirements and needs, there are a few paths you can take:

• Build your own server-side A/B testing framework in-house
• Integrate an open source solution
• Invest in a commercial experimentation platform like Optimizely Full Stack

When considering your options, considering these four areas can help you get informed and determine which solution makes the most sense.

1. Statistical rigor: Do you have confidence in the system and the results it produces?

When implemented correctly, experimentation can help teams quickly identify which features and changes will drive business value, and which will fail. The best teams embrace failure, and try to fail as fast as possible so that they can move on to ideas that work. To fail faster and learn quickly, your teams need to trust that tests are being run correctly, and that the results are accurate.

However, many companies underestimate the difficulty of collecting data reliably and maintaining their analytics pipelines over time. When events aren’t tracked correctly or analytics integrations stop working, it results in delays and slower experimentation velocity.

In many organizations, the responsibility for setting a sample size and analyzing results typically falls on an analytics or data science team. Some analytics teams who are responsible for analyzing every test find that they quickly become the bottleneck as experimentation scales across multiple teams.

When evaluating a platform for your company, ensure that your analyst team is involved to determine which solution makes it easiest for them to quickly access data and communicate results to stakeholders.

2. Ease of Use: Is the system easy for developers and accessible to business users?

The best teams in the world run thousands of experiments each year to maximize opportunities to learn. Usability for both technical and non-technical users can be the difference between running several experiments in a year and running hundreds or even thousands.

On most product teams, developers will be implementing tests directly within your codebase, so they often make the initial decision about which solution to start with. Many open source tools or homegrown systems operate entirely within code at the beginning. This means that a deploy is required for each new change to the experiment, and that it’s difficult for product managers or analysts to understand what’s happening with running experiments without pulling data out of a database and analyzing reports by hand.

When determining whether a system will be accessible to business users, identify whether it contains remote configuration tools and easy-to-understand UI design. These features help teams increase experimentation velocity by decoupling experiments from deployment, and providing transparency into experiments across the organization.

When it comes to developer productivity, focus on features of the solution like robust documentation and support for multiple languages that will enable the engineering team to spend less time figuring out how to run tests, and more time working on customer-facing feature work. A robust solution will also include APIs that can be used to automate certain task or integrate more deeply into development teams workflows.

3. Total Cost of Ownership: How will the system be developed and maintained over time?

Building in-house or adopting an open source framework typically comes with a relatively small upfront investment. Over time, however, additional features and customizations become necessary as more client-side and server-side teams use the platform, and maintenance burdens begin to distract engineers from core product focus.

To avoid unforeseen maintenance cost, ensure that in addition to understanding your team’s initial set of feature requirements, you’ve budgeted time for bug fixes and considered the more complex features you might need down the road. Will you need to run experiments that are mutually exclusive, or use advanced targeting features? Determine how custom these advanced features will be, and estimate the resources required for building and maintaining them. Anticipating your organizational bandwidth for engineering and maintenance can help you plan beyond initial development, inform your decision to build or buy, and ensure critical features are available when your team needs them most.

4. People and Process: Who will evangelize experimentation culture across your company?

Successful experimentation is as much about culture as it is about technology. The top teams invest in people and process to ensure that cultural change takes root, and that data-driven decisionmaking becomes the norm.

Whether you decide to build an experimentation platform in-house, or invest in a commercial solution, it’s critical to have the right people and processes in place. Identifying the members of your organization who will be responsible for defining key metrics and championing a culture of experimentation company-wide can help ensure you get the most out of whatever solution you choose.

For more insights into whether to build or buy an experimentation platform, and information to help you determine which course of action is right for your organization, download the Build vs. Buy: Implementing the Right Experimentation Solution whitepaper.