6 Experimentation secrets from Airbnb and Uber

Last week we hosted over 130 experimenters at Optimizely HQ in San Francisco for our first live event of the year. We were lucky enough to be joined by two brands — Airbnb and Uber — that have fundamentally changed the way we travel and get around today:

Panelists
Jami Schwartz, Head of Marketing, Airbnb for Work
Caitlin Anderson, Senior Marketing Manager, Uber
Claire Vo, SVP of Product, Optimizely

For those of you that couldn’t make it, we’ve put together six highlights from the night:

1. New Hires = New Ideas

Scaling an experimentation program often means increasing the volume and the scope of your tests. So, how do teams continue to get new and creative test ideas? There’s analytics, heatmaps, user research, and many other methods. But for Jami Schwartz at Airbnb she turns to interviewing new employees: “I always like it when people who have just started at the company give us new ideas, because they’re the ones who have no context and haven’t really thought about the different stages of the website yet.” Both Uber and Airbnb collect ideas from across the company. Uber’s Cait Anderson shared that “Uber also collects internal idea submissions. In addition, we do ideation sessions with the team and our primary partners, and then prioritize those to set out what our schedule and our commitments are.” So, if you want new test ideas, start talking to your new hires.

2. The Modern Marketing Stack

Marketers are always hungry to learn about new software that they can add to their toolkit. We know it’s not enough to simply have website analytics in place. So what else is in Airbnb and Uber’s marketing stack? What are the tools that they use to feed their experimentation program? Here’s what they shared:

Airbnb: “Our stack is pretty significant. We obviously use Google Analytics as a single source of truth. And that is a company wide tool as well. We use HotJar for heat mapping. We use Marketo for a lot of our CRM needs. Our whole marketing tech stack comprises of over 20 pieces. But I would say those are some of the top ones that come to mind. And then User Testing is something that’s on our roadmap that we really want to use.”

Uber: “We also use HotJar. It is our secondary tech tool that we use for story metrics. We have one KPI for all of our tests that are launched, and that KPI is dependent on a product. But our story metrics are really imperative. And then internally we have dashboards built. We don’t get to the visualization aspect yet, and that will come, but the dashboard helps us instead of fielding requests and reports, we’re able to see in real-time what’s happening.”

As you can see, marketing tech stacks can vary, but ensuring you have the software and people to analyze the behavior on your site is important to successful decisions.

3. That Aha Testing Moment!

When asked about the biggest ‘aha’ moment they’ve had in their experimentation journey, Uber shared a great story. This story is about remembering that your audiences always matter. And, that you should continue to stay customer-obsessed. Cait Anderson explained, “Each customer is different. So one of our biggest busts that we thought would be the biggest winner, was that we thought we could scale an incentive program for drivers from the organic world to the paid world —  and that our visitors would act the same. And then to layer in complications, we also thought that a brand update may not make a difference. But it did make a difference and you can’t just simply scale a win anywhere you want and see a revenue boost.” This was a great example of not relying on our own hunches or best judgement, but instead, actually using data to uncover differences in customer behavior.

4. Uber’s Criteria for Scaling Experimentation

We all have dreams of scaling up experimentation. Whether that’s going from a handful, to a thousand, to tens of thousands like we hear Netflix does. So, we asked Uber, a company where experimentation has been adopted across a global team, how they scale their practice. We wanted to know, how has that adoption process been? Has it been even across all teams? Here’s what Caitlin said:

“Maturity of market is one key factor, and then the second factor is the size or amount of traffic. We can’t reach a segment if we don’t have enough traffic and we aren’t changing our KPI. Uber’s marketing team has one KPI that we’re always trying to hit. So, scalability at a global level is based on the maturity level of that international market, what the product offerings are in that market, if users are going to the web experience, and then what the traffic level of the web experience is.”

5. How We Hire (aka What Makes a Good Experimenter?)

What is needed to be a great digital experimenter today? Cait Anderson believes that being process-oriented is key. “The job is still very human and high touch, unfortunately in a tech world, which is kind of shocking. So that would be the number one trait (being highly process-oriented). An analytical skill set, is great. And then customer obsession. I would say those are the three skills.” Jami also went on to echo the point on customer obsession. “You should be out in the field. And I think as marketers, we don’t get that opportunity. We have field marketing of course, but I encourage people to get on calls with sales, get on the phone, field your own sales calls, essentially, because I think that’s critical. I actually think it’s probably the necessary thing to be good at for any marketing role.” What about technical skill sets? Both Cait and Jami were less worried about technical skills since they said many hires have “entrepreneurial spirit” which means they have the drive to learn and pick up new skills on the job.

6. Stepping Outside our Silicon Valley Bubble

“We’re all in San Francisco and we work at these big brands, but believe it or not, there are people who’ve never heard of Airbnb. And what you start to realize is, oh, someone comes to your site and they actually don’t know what Airbnb is. So you’re serving all this bottom of funnel content and you’re thinking about how can we get them to convert and you’re not even thinking about this level of awareness. So I think qualitative data is super important. We have forms on our site, so I read the comments weekly, and I’m always shocked at people asking, “What’s Airbnb? How would I use this? Do I book it…” Jami went on to share that it’s important to step back and listen to your customers when thinking of new test ideas. Airbnb looks at where these visitors are coming from (organic vs. social channels) which can give them a hint as to what level of base knowledge they’ll have about the brand.

For those that attended our event last week, thank you for coming out to join us — we hope you walked away feeling energized to scale your experimentation efforts in 2019. And for those that couldn’t make it, we’ll see you next time.

If you’re wanting to see more examples of the world’s best brands using Optimizely to improve their digital experiences, check out our new eBook titled, Digital Change Agents: Lessons, Tactics, and Stories from World-Class Experimenters. It features insights and test ideas from the teams at Atlassian, Visa, Sky, and more.