The Impact of Symmetry in Online Marketing

TL;DR

• Visitors who saw the symmetric version of our PPC landing page converted to sales leads more frequently than users who saw the non-symmetric version, with high confidence.
• The observed improvement in conversions from visitors to sales leads was more than 39%.
• This symmetry experiment saved Optimizely hundreds of thousands of dollars and led to tens of thousands of additional sales leads.

Background

You may have heard that keeping your ad consistent with your landing page is good. But is that a myth? And if it isn’t, how can you quantify the impact of this symmetry? In early 2013, we performed an experiment on our Pay Per Click landing page aimed at answering these questions. Our goal was more to answer an interesting question than to find a conversion lift, but we were pleasantly surprised to discover that making our landing page mirror our ads led to a tremendous lift in conversions.

The experiment

We send traffic to our PPC Landing Page from lots of advertising campaigns, including search marketing campaigns. In some of our SEM ads we call ourselves an “A/B testing tool” and in others we call ourselves a “Split testing tool” and so on. These are all valid ways to describe Optimizely, but some people think about our space differently.

In the control, the headline of the page was always the same. It always said “Test it out for free,” no matter which ad the user had clicked on.

In the variation, the headline of the page mirrored the search ad. For example, if the search ad said “A/B testing tool” then the headline on the landing page would say “A/B testing tool,” and if the search ad said “Split testing tool” then the headline would say “Split Testing Tool.”

Original vs. Variation

Those are just a few examples, but conceptually this is how we split the traffic between control and variation.

Experiment results

We saw a 39.1% increase in conversions from visitors to leads in the variation with symmetric messaging. In the control, which had non-symmetric messaging, the conversion rate was 12.21%. But when the headline on the landing page was symmetric with the ad copy, that conversion rate jumped to 16.99%.

Funnel impact

We get a sales lead whenever someone submits a form on our PPC landing page, so the 39.1% increase in form submissions translated to a proportional increase in sales leads. We didn’t pay a penny more for the 39.1% increase in sales leads. In fact we paid slightly less because the experiment reduced the bounce rate on the landing page, thereby improving our Quality Score in AdWords and giving us a discount on a CPA basis.

We came into this experiment hoping to answer an interesting question and ended up getting tens of thousands of additional leads at no added cost — leads that would have cost us in the hundreds of thousands of dollars had we not run this test. (We keep our marketing spend, lead numbers, and cost per lead data pretty close to the vest, so please forgive us for the vagueness here!)

Common Questions

Q. What exactly do you mean by symmetry?
A. There are different kinds of symmetry, but in general two things are symmetric if there is something you can do to them that leaves them unchanged (source). In this case, what makes the landing page and the ad copy symmetric is that you can replace the headline on the landing page with the headline in the ad text and they will remain the same.

Q. Did you repeat the test and notice similar results?
A. Yes, we repeated the test and noticed similar results — not only on this landing page, but in other parts of the marketing funnel as well. We also have a long-term A/A test on this page to help detect any potential anomalies in the data.

#BBD0E0 »