Your Email Open Rate is High? That’s Nice.

Email is a core part of online marketing because it works.

The volume of marketing email I receive (therefore, unsubscribe to) has increased a lot over the past year and will probably continue to do so. 59% of marketers plan to increase their email marketing budgets in 2015 (source: Salesforce Pardot). Lucky for us marketers, we have tools to help cope: Unroll.me to de-clutter the deluge of webinar invites, and A/B testing to help make the emails we send perform better.

I want to focus on the latter: the A/B testing of the emails. There’s a widely held belief that open rate is an important success metric. While this is true, it is incomplete. The open rate is only as good as the conversion rate it produces.

A high open rate is nice. It’s an important signal you’re writing evocative subject lines and optimizing your send time. But if people don’t take action or convert on that email they opened, then the act of opening is much less meaningful. Getting maximum value out of A/B testing marketing email requires optimizing for key metrics that really matter to the business—and open rate is not one of them.

In this post, we’ll cover:

1. The testable parts of an email (besides the subject line)
2. How to decide which part to test based your key goals (open rate is not one of them)
3. Ways to execute your tests (the 50-50 vs. 10-10-80)

The testable parts of an email

The email itself has 3 parts and each part has numerous testable elements:

1. The envelope: The up-front information that explains who the email is from and what it’s about.
2. The body: The content.
3. The landing page: The place the email directs to.

We’ll cover this later, but worth noting that just because you can test all of these elements doesn’t mean you should. Being methodical and deliberate about only testing hypotheses that relate to your big company goals will help you drive more value from your email marketing programs. (Here are some of the lessons we learned from email A/B tests last year.)

Testable parts of the envelope

Testable elements of the email envelope are the subject line, the snippet/pre-header text, the from name, and the from email.

Tip from an email marketing expert: “People worry about subject line a lot: Can you use exclamation points? How long is too long? Be bold and don’t shy away from anything,” says Jessica Langensand a Marketing Programs Manager at Marketo. “Everything is fair game.”

Testable parts of the body

Testable elements of the email body are the headline, the images, the body test, the call-to-action text, and the layout of the entire email template.

All parts are connected: The envelope copy should relate to the body copy, which should connect to the headline and action on the landing page. Reuse images from the email to reinforce congruency. Reuse headlines. The more you make people think, the worse your conversion rates will be. Most of the time, good flow means keeping things really simple and obvious.

Testable parts of the landing page

Testable elements of the email landing page are the specific landing page you direct to, the headline on that page, the copy and the call to action.

Tips from an email marketing expert: “The biggest crime is sending people to your homepage from email,” says Chad White a Lead Research Analyst at Salesforce ExactTarget Marketing Cloud. “Don’t make visitors work. Hold their hand.” 

A word to the wise as a marketer who receives and writes a lot of marketing email: don’t forget that the landing page is part of the email.

Deciding which part to test

Like we said, you can test just about anything related to email (including time of day, day of the week). But withhold the urge to test everything and instead prioritize tests that will impact key company goals. What are key company goals? Think metrics like customer retention, product usage and average revenue per visitor.

Randomly testing subject lines or calls to action may increase your open or click-through rate, but these are not the metrics the CEO cares about. That being said, testing something is more helpful than testing nothing. So if you can test subject lines, but don’t have the bandwidth to think about measuring further down the funnel, test all the subject lines you desire.

Here is a framework for planning email A/B tests that map to company goals: 

• Start every test by identifying your macro company goals and the micro goals email can help impact. 
  • An example from Optimizely: Turning more people who sign up for Optimizely into avid users is a goal my team is focused on. This is a massive goal. To achieve it, we have to break it down into parts. So we ask: what are the steps someone must take to use Optimizely? They all need to copy and paste a line of code (snippet) onto their website. Ding ding! We have an answer and a micro goal for our email: get more people to install the snippet.
• Find every email that contributes to this goal. Which emails in the database help achieve this? Make a list of them.
  • Tip: Pick the emails that reach the most people. Testing emails that reach a large audience reach statistical significance faster. Good candidates are behavior-based trigger emails like a welcome email. Or emails in nurture campaigns. These emails are great places to focus testing. Focus on just one email for now.
  • Based on our micro goal, we chose to optimize the email that triggers to people who have not installed the snippet.
• Decide what you want to learn and write your hypothesis. Tests that have a sound hypothesis will teach you more than those that don’t.
  • We decided to test aspirational vs. directional messaging in the email body. Our hypothesis: An email that explains what they’re missing by not installing the snippet will perform better than one with directions on how to install the snippet.
• Decide what success looks like. Be very specific. How are you measuring this?
  • We’re measuring success by conversions — the number of people who install the snippet after receiving the email, not by clicks or opens.
• Set up the test and let it run.
• Record the results. Keep all of the hypotheses, variations, and results in one place. We use one at Optimizely to keep track of every test. We refer back to it often, especially when we’re writing new emails.

Be systematic about testing. Identify low performing areas where you can achieve fast gains and then commit to a testing calendar. Come up with a list of hypotheses you have and design tests around them. Hopefully, some tests will produce grand slams. Quite often though it takes many tests to find one that drives a meaningful difference between two variations. Don’t be discouraged!

For email inspiration and hypotheses, subscribe to emails from other brands for inspiration and bookmark reallygoodemails.com.

Ways to execute your tests

Finally, there are different styles of executing email A/B tests and each should be used strategically for different reasons.

The 50-50

What is it?

A style of test where you send version A to 50% of your list and version B to the other 50% of your list.

When should you use it?

Use a 50-50 test when you’re testing emails you send frequently such as newsletters, daily/weekly/monthly digests, product announcements, or sale alerts. Use this type of test when you are looking for results that you will reuse again and again.

The 10-10-80

What is it?

Another style of test when you send A to 10% of your list, B to 10% get a confident result and send the winner to the remaining 80% of list.

When should you use it?

The 10-10-80 is useful whenever you want to test an element of an email that’s part of a time sensitive campaign, AND you have a very large list.

Say you’re planning a marketing campaign with 5 emails that you’ll send to your entire list. This is the perfect scenario for a 10-10-80 because you want to get quick results on which elements are performing best so you can apply those lessons on the remaining campaign emails. This way, you will capture as much value possible from the campaign.

 A recap: things to remember when setting up email A/B tests

• Run tests that map to your company goals. Open rate is not a company goal.
• The landing page you direct people to is equally important as the email itself. Opening is important but taking a valuable action from the email matters more.
• Decide how you’re measuring success before you run the test! If you dont, it will be really hard to make sense of the results.
• Keep a record of everything you test so you can reflect and constantly improve.