Test and learn your way to product greatness

What it Means to Test and Learn

For teams designing, building and analyzing software products, the Test and Learn movement represents a shift from launching new functionality and hoping for the best. The old model looks like this: DESIGN > BUILD > LAUNCH PRAY 🤞

Test and Learn is about empowering teams to challenge assumptions, bring in customer feedback much earlier, and continuously improve and validate products with real users and experiences. The new model looks more like this:

DESIGN > BUILD > EXPERIMENT > ITERATE ✌️

Companies like Netflix, Google and Amazon did not just stumble upon product greatness. They take a purposeful and disciplined approach to building habit-forming products. And they do it with data and testing.

Applying the Methodology

There are many different ways to put Test and Learn into action. A few months ago, Optimizely hosted a half-day virtual summit where experts led in-depth sessions from Dropbox, WSJ, Indeed, Booking.com and more. We’ve captured this wealth of information in the Test and Learn ebook. The speakers shared transferrable frameworks for achieving success in design, engineering, data analysis, and product using experimentation.

Nir Eyal, author of Hooked: How to Build Habit-Forming Products and the master class keynote speaker, unpacked the human psychology that drives the habitual use of the products we use. From Google’s productivity suite to Facebook and Online Games, Nir shares his research dissecting how these products work and play into our natural tendencies and desires.

He also talks about the responsibility of the teams that create these products to make sure they are positively impacting users at scale. Since the virtual summit, Nir has published a new book, Indistractable, about supporting balance, focus and healthy relationships in our lives as consumers. Read this New York Times article to learn more about his latest work.

The Results

Putting Test and Learn into action yields tangible rewards. Dropbox redesigned subscription workflows as an experiment, resulting in a better user experience and business outcomes. Through testing, they drove a massive increase in traffic to the upsell page which ultimately led to a 3x increase in subscription upgrades.

WSJ did extensive research on which early usage habits led to customer retention in order to determine what to optimize for.

They found that new users who downloaded the mobile app were much more likely to become habitual, tenured users. By running multiple product experiments, they significantly increased the share of new users downloading the app.

Applying a Test and Learn approach to code ownership through feature flagging, led to more observability and resilience at Honeycomb.io. And when Indeed.com looks at metrics they use experiments to deliver long-term value for their customers.