Experimentation platform: Should you build or buy?

Having invented the category of digital experimentation in 2013, Optimizely has more experimentation customers and more experiments run on our platform than anyone else. We sometimes see our larger customers leave us to build an internal experimentation platform. We’ve also seen a good number of these customers come back to us a few years later. 

In addition to this historical knowledge base, we recently interviewed 15+ large experimenting organizations that are current customers, former customers, returning customers, or non-customers all of whom have built or considered building an internal experimentation tool.  

Before you roll your eyes and click away, hear us out. We promise to give you the full picture of the best option for your business. Let's dive into the key considerations, drawing insights from Optimizely's extensive experience in the field - both the triumphs and the face-palms.

TL;DR 

• Companies that build platforms rarely approach it with the methodical rigor the decision deserves.
• Customers who build their own system generally spend more to get less, but there are valid justifications. 
• There’s a right way to make the decision, and Optimizely has the expertise to empower any organization to make the decision well for its own purposes.

The foundational principle of build vs. buy decisions

All companies are barraged with questions of whether to buy or build things. Much has been written about it. The core framework is pretty simple:

• Build only what is core to your customer-facing product or where it creates a significant competitive advantage. 
• Buy everything else.

Optimizely tries to apply that principle just as much as anyone else: despite being a technology company, we have hundreds of technology vendors ourselves. 

Why do companies think they should build an internal platform?

The idea of a custom-built platform may be enticing for companies because of following reasons:

1. Engineering’s instinct to build

Engineers are the builders of digital experiences. They are the most important teams for digital success. Naturally, their initial instinct is often to build. Every company we talked to acknowledged that the decision to make an in-house platform began with an engineer who wanted to build an in-house solution.

2. Perception of cost savings

Once the costs of a 3rd party platform rise above a certain level, organizations have an instinctive reaction to the cost and believe they can save money by building an internal tool.

3. Amazon did it that way

There is a small group of massive tech companies with market caps over $250b, unique cultures, and the largest number of top engineers who have built good internal tools. When someone from those companies joins a slightly smaller or less technically sophisticated organization, they often assume the playbook is the same. 
 
The in-house experimentation platform struggles to achieve intended cost, velocity, and capability goals.

4. Cost

The decision to build is rarely justified with robust cost-modeling. This is further compounded by the lack of comprehensive requirements gathering and an end-to-end plan. Because the idea usually begins with engineers, they underestimate the non-engineering needs and processes that are critical to high experiment velocity, even in the most sophisticated, large organizations. This applies first to build costs upfront and then to ongoing maintenance as well.

We’ve engaged countless prospects or customers who are thinking about building a platform. We’ve built a robust, unbiased cost model that we complete with the organization. Every time, it delivers a variation on the same message: an internal platform costs more to get less. This is because Optimizely has had hundreds of engineers building our experimentation platform for over 10 years. There is simply no way to get around all that time and effort.

5. Velocity

Experimentation ROI is a factor of quality and velocity. We’ll set experiment quality aside for the purpose of this exercise and assume that all the experiments being run are the ‘right ones’ to run. You need velocity because what experimentation exposes is that even the smartest companies can only guess the best digital experience 10-20% of the time.

Source: Lessons learned from 127k experiments

If you are only going to get 1-2 out of 10 bets right, you need to make a lot of bets!

We consistently find that even with existing in-house tools at $100B+ companies that the velocity is a lot lower than what it could be with Optimizely. 

What causes the velocity drop?

When companies build internal tools they don’t collect comprehensive requirements. Because the idea usually begins with engineers, their definition of a functional experimentation platform is a black box that can run experiments, but which is entirely dependent on expensive engineering hours to do even basic tasks related to experimentation with no visibility or ability for non-engineers to do anything. 

1. No interface and UX

   They underestimate the time and efficiency costs of an easy-to-use interface for all parties, including engineers.

   • New engineers can’t figure out how to run experiments so it’s harder for experimentation to scale. 
   • Stakeholders who may do a lot of the planning work like marketers or product managers can do no work related to experimentation. Many small, non-technical tasks must be done repetitively by engineers.

2. Experiment run time

   Separate from the quality of the statistical algorithm, because the program is a black box, even the act of checking on an experiment is a large lift. If that large lift is performed, it may result in the finding that the test hasn’t reached significance, which means a task that is already expensive to perform just once, has now been done multiple times per experiment. Some internal systems ‘solve’ this problem by setting pre-determined test run times, but this is wasteful in many use cases if the test reaches statistical significance early.

3. Data analysis

   Even simple test analysis requires multiple, cross-platform steps by analysts. In the Optimizely platform it takes 5 minutes to look at our results page. 

   

4. Data accuracy

   Consistently randomizing audience groups and then maintaining consistent bucketing across visits or platforms is hard to do. We’ve had multiple large customers who left us to build their own platform only to return a few years later because they ran 100+ A/A tests and could never deliver credible results. Nobody wants to run tests when they don’t trust the data!

5. Governance + Visibility

   Because these systems are often black boxes only alterable by engineers, there is a higher likelihood for erroneous deployments or broken variations to stay live for longer. When something goes wrong, because nobody can see what’s going on, it’s harder to diagnose and the platform and discipline lose credibility in the organization quickly.

6. Documentation

   Any technology requires lots of documentation to enable more teams. Keeping those documents up to date is an incredible amount of work that they don’t have time to do. 

How this plays over time?

While there may be alignment when the plan to build an in-house platform is initially formed, because the project is rarely fully scoped, the time-to-value extends beyond everyone’s expectations. This often results in the available resources declining before the project achieves the level of functionality to deliver on the promises.  

When is building a good idea?

While buying is often the best choice, there are scenarios where building an in-house platform could be justified:

• You are Amazon: There is a very small number of companies with market caps north of $250B, high technical expertise, and unique cultures that had experimentation so deeply in their DNA from day 1 that they invested the right amount, at the right time to build a good internal platform and actually have some credible competitive advantages because of it.
• Existential data risk + scrutiny: For that small number of companies in that $250B+ market cap bracket, especially with previous data-related breaches, scandals, or government investigations, it may be worth it to be able to say that no 3rd party ever has access to any 1st party data.
• Hybrid approaches: Some of our customers want to build some components of the experimentation platform and buy others. The circumstances are too diverse to capture broad trends, but this can be a mid-point option. 

How to make the right decision for your organization

The buy vs. build debate isn't one-size-fits-all. To start, conduct a thorough analysis. Perform a comprehensive cost-benefit analysis, factoring in not just initial development costs but also ongoing maintenance, opportunity costs, and potential limitations.

For reference, Optimizely commissioned Forrester Consulting to conduct a Total Economic Impact™ (TEI) study and demonstrate the potential financial impact of adopting Optimizely. What more, you ask?

• Assess core competencies: Ask yourself whether building an experimentation platform aligns with your company's primary focus and competitive advantages. But how do you go about finding the right platform for your needs? Our experimentation RFP template shows the questions you should be asking.
• Consider long-term implications: Think beyond immediate needs and evaluate how your choice will impact scalability, flexibility, and resource allocation in the years to come. For example, your experiments, need to reach statistical significance. Optimizely's sample size calculator ensures you no longer need to wait for a pre-set sample size to ensure the validity of your results.
• Seek expert advice: Consult with experienced practitioners and vendors in the field. If you have a platform or are thinking of building one, come to us! We’ll do a buy vs. build total-cost-of-ownership and velocity evaluation for you.

Optimizely can help...

1. Optimizely customers: If you are building an in-house platform or someone in your organization is thinking of building one: please reach out to your Customer Success Manager immediately or me (pete.gray@optimizely) and we will complete a total cost of ownership study free of charge. We truly want what’s best for you: let us help you evaluate this decision properly. Remember, the best part about our advice: it’s free, it’s informed, and you can do whatever you want with it! 
    
2. Non-Optimizely customers: If you have a platform, or are thinking of building one, come to us! We’ll do a buy vs. build total-cost-of-ownership and velocity evaluation for you!