Organized Ideation: How Hotwire Runs 120+ Experiments Per Year

This is the first post in a three-part series about the optimization program at Hotwire, a leading travel booking site owned by Expedia, Inc. In the series of three posts, we dive into the optimized testing process of Pauline Marol, the Lead Product Manager for Site Optimization at Hotwire.

Pauline Marol is the Lead Product Manager for Site Optimization at Hotwire, a travel booking business owned by Expedia

Meet Pauline

After poring over hundreds of potential experiment ideas in an Excel spreadsheet, rating each one by potential impact, and sorting the backlog of submissions, Pauline is armed with a prioritized list of experiment ideas, ready to be built.

She pauses, mid-conversation, and declares: “I have the best job in the world.”

Pauline is the Lead Product Manager for Site Optimization at Hotwire. Her meticulous method of experiment prioritization is just one of many pieces of evidence of a strong culture of optimization at her company. She leads a testing program supported by a team of more than seven designers and developers. They are chartered with improving conversion rates on Hotwire.com’s desktop and mobile web properties by running over 120 experiments each year. This task has merited one of the company’s four KPIs and CEO sponsorship.

She’s an advocate for A/B testing and optimization both on and off the clock—she’s launching a “Test and Learn” program at Hotwire that’s dedicated to sharing best practices on optimization across all teams that have the ability to test. She also leads Optimizely’s San Francisco user group.

“The travel industry is extremely competitive—everyone is testing like crazy,” says Pauline. “I have dedicated time on my calendar to keep up with what’s going on in the industry so that we can continue to develop the best online experiences for Hotwire customers.”

Hotwire, owned by Expedia, Inc., provides hotel bookings, car rentals, and airfare to customers through a variety of digital experiences, including the web and native mobile apps. To improve bookings and customer retention, Pauline focuses on increasing both the quality and quantity of experiments her team is able to run.

How Hotwire Produces High-Quality Ideas to Test with “Conversion Veins”

Hotwire knows that the best optimization ideas come from their team members and customers. To support a well-rounded brainstorming process that incorporates input from many perspectives, Pauline draws on the experience of customer-facing teams, learnings from other Expedia, Inc. domains (Hotels.com and CarRentals.com), and the User Research team.

Pauline also orchestrated a company-wide call for ideas: Hotwire’s Big Test Challenge. The contest encouraged Hotwire employees submit their ideas for testing with gift cards and other incentives.

To help start the flow of creative ideas, Pauline helps her audiences classify their ideas into different ‘conversion veins’—thematic optimization areas that can affect the conversion rate and overall customer experience.

There are eight conversion veins that Pauline recommends to her audience to inspire and capture high-value experiment ideas.

8 Experiments that Stemmed from ‘Conversion Veins’

Each of the examples below shows an experiment created from brainstorming within the conversion vein. The experiment variations are all winners that are now live on Hotwire.com.

1. Social signal: Sometimes called social proof, these indicators show that a listing is popular or recommended by other travelers. These signals could include ratings, reviews, testimonials, or other iconography.

Control	The control experience displayed a “% recommended” rate, a search result feature that fits within the Confidence conversion vein.
Winning Variation	In this experiment, introducing the word “Great” added context to whether or not the recommendation rate was exceptional, average, or below average.

2. Confidence: Any change that can improve customer understanding, peace of mind, or assure the visitor they’re getting a competitive price. Improving confidence can happen at any stage in the booking funnel, from signup to search, the listing pages, badging and side rails, or checkout pages.

Control	Winning Variation
No Confidence Module	This sidebar content highlights the process for securing discounted rates through Hotwire, a process that may seem opaque to visitors during their search process.

3. Price display: Changes to how prices of flights, hotels, or car rentals appear in search results, during checkout, or elsewhere in the customer experience. The team might experiment with how discounts are displayed or highlighting free amenities to increase perceived value.

Control	The control experience displays the discounted price on the left as part of the search results.
Winning Variation	The variation experience adds a % discount and also highlights additional amenities to increase the perceived value of the search result.

4. Navigation: Experiments that affect how visitors search and consume information while researching and booking travel. This could include filters available to sort search results, navigation elements at different stages of the funnel, or how users proceed through their checkout process.

Control	The control experience includes a “back to search results” navigation.
Winning Variation	This experiment added a “Jump to:” navigation at the top of hotel pages to help visitors find relevant pieces of information about the listing quickly.

5. Simplicity: Any unnecessary elements that can be removed as part of the experiment can potentially improve the customer experience and improve the ultimate conversion rate.

6. Product info layout: Hotwire’s customers travel for a variety of reasons and have wide-ranging needs when they’re away from home. Is it important that a hotel has free internet, or is pet-friendly? What about parking or a fitness center? Now, is it best to showcase this information at the search results level, or on the listing page? What is the most helpful layout? All of these choices can be tested.

Control	Winning Variation
The control experience shows amenities as icons of mixed colors below the CTA.	The variation experience spaces out the icons, adds a description of each one, and a right-hand navigation element to view more.

7. Visual imagery: There are few activities that inspire emotion quite like travel. Visual imagery is a powerful way to bring the emotion of travel into the customer experience—but does it help or hurt conversions? What’s the right balance of visuals to information and functionality for Hotwire?

Control	Winning Variation
The control experience shows car options in a baseline gray color.	The variation experience swaps out gray for colorful cars, which increased the performance of the experience.

8. Urgency: Travel borrows from its online purchasing cousin, e-commerce, in these types of experiments. What time-sensitive cues can encourage visitors to move from researching to purchasing? Countdown clocks, limited inventory, or a limited-time offer?

Control	No Urgency Notification
Winning Variation	This experiment added a “Jump to:” navigation at the top of hotel pages to help visitors find relevant pieces of information about the listing quickly.

Each of these conversion veins has a strong basis in consumer psychology and understanding of UX/UI design principles. With these pillars of testing ideation at her disposal, Pauline is able to aggregate a wide array of experiment ideas and start the process of evaluating their potential.

Stay tuned for next week, when we’ll reveal Pauline’s prioritization process and tips for how to remain unbiased during the testing process.