Case Study: Testing Personalized Experiences at Secret Escapes

In part one we covered how Secret Escapes used Optimizely to experiment on their iOS app onboarding flow, double signup rate, and generate positive ROI from paid mobile marketing.

In this post, we’ll dig into another series of experiments that are driving continual increases in new user signups and revenue. In both experiments, Secret Escapes is testing the impact of personalized experiences.

Experiment 1: Personalization for PPC Landing Pages

The company had high click-through rates on their search marketing campaign ads and directed all of this traffic to a single Pay Per Click landing page. They saw an opportunity to convert more paid traffic on this page.

Tom Evans, a product manager and A/B testing lead at Secret Escapes, hypothesized that mirroring copy on the landing page with the copy on the ad would improve conversion rates. In other words, they tested the impact symmetric messaging between ads and landing pages had on the rate of new user signups.

The test stems from the idea of matching user intent with expectation. Entering a search in Google is a clear display of user intent. The hypothesis is that a symmetric experience from the ad to landing page —one that clearly matches that search intent — will make people more likely to take the desired action on the page.

Secret Escapes “spa” PPC ad in Google.

Tom ran this test for their “spa” AdWords group first. He tested two different personalized pages against the control landing page with no personalized elements:

In variation 1, he changed the image to a beautiful spa pool and changed the sub-head to include the keyword, “spa”.

In variation two, he kept the beautiful spa image and made the headline copy symmetrical with the ad.

The test ran on a sub-set of traffic: users who came in using spa keywords. In Optimizely, he set this up by targeting the test to a query string used by a number of the spa ad-groups. View variations side-by-side.

Experiment results

Secret Escapes saw a 32% increase in conversions in new user signups from visitors who received the symmetrical messaging in variation 2, the spa image and symmetrical headline.

Rather than a one-off, this result translates into a proportional increase in new users for Secret Escapes at no added cost of AdWords spend. Next, the team will test symmetric messaging on different ad groups.

Experiment 2: Personalization for Logged-In Users

Using search data to personalize vacation listings

For a logged-in user, visiting the homepage (secretescapes.com) used to be a long list of vacation listings to everywhere from a casino in Las Vegas to a beach in the Dominican Republic. Other than timeliness of the deal, there was no rhyme or reason behind why a particular listing appeared.

Original logged-in homepage experience for Secret Escapes.

“Whilst we curate and hand-pick every offer that features on the site, we don’t do a great job at creating a personal list for our users,” Tom wrote in a post on the Secret Escapes’ tech blog.

Not anymore. After running an experiment that increased conversions, it’s more likely that logged-in users will see homepage listings that are tailored to the types of vacations they have searched for previously.

“Personalisation is one of the things I really expect to blow up in 2015,” he continued. “With so much data lying around about our customers, and their behaviour, we now have a great opportunity to optimise their experience by providing more relevant sales and information.”