Guide to A/B Testing Campaign Emails

read
Last updated at:

Overview

Klaviyo's A/B testing feature for campaigns allows you to easily test different subject lines and content to refine your messaging and optimize your send times. A/B testing is currently only available for emails. 

This article covers how to set up and run a campaign A/B test, how to read the results of an A/B test, and a few use cases for A/B testing campaigns.

How to A/B Test Campaign Emails

Navigate to Campaigns > Create Campaign > Email.

Name your campaign and select the lists or segments you want to send to. 

Click Continue to Content

Input the subject line and, if desired, edit the preview text, sender name, and sender email address. 

The email edit screen, showing where you can add the subject line, preview text, and sender name and email.

Next, create the first version of your email, adding your text, images, and links. When you’re done, click Save Content

At the bottom of the page, click Create A/B Test

The Create A/B test button is located at the bottom of the email preview screen.

This will automatically create a second, identical variation of your campaign and bring you to an A/B testing page.

The A/B Test Settings page.

Test Content

On this page, edit one of the variations for your A/B test. It’s important for A/B tests to compare one factor at a time, so if you edit the subject line, do not change anything else. 

Clone a Variation

If desired, you can add more variations by clicking the Clone button, although we recommend sticking to just two variations. 

The Clone option, wbich is available for both variations.

Test Send Times

You can also test send times to determine when your audience wants to hear from you. 

To do so, click Switch to testing send times at the top of the page.

In this case, keep the content of both variations exactly the same. Then, choose the times you want to test. 

The testing send times option is located in the upper right of the A/B test Settings page.

If you don't know where to start, we recommend trying an evening time. Alternatively, if you qualify, you can use Smart Send Time instead. 

Configure A/B Test Settings

After creating your content, decide the size of your testing pool and the length of your testing period.

First, choose the winning metric, which is the main metric you want to see improve. The default is open rate, but you can switch it to click rate.

  • Open rate is best if you are testing differences in:
    • Subject line
    • Preview text
    • Sender name/email address
  • Click rate is best if you are testing differences in the body of the email

9/20/21 Update: with the release of iOS 15, macOS Monterey, iPadOS 15, and WatchOS 8, Apple will change the way that we receive open rate data on your emails by prefetching our tracking pixel. With this change, it’s important to understand that open rates will be inflated. It may be useful to look at the number of iOS-specific users within your data to understand how your rates may change.

With regard to A/B testing, Klaviyo should account for these inflated open rates; however, you may need a higher threshold to reach statistical significance. If you regularly conduct A/B testing AND more than around 45% of recipients open emails via Apple Mail, you might want to consider switching to click rates as your primary metric for testing.

For information on ways to potentially mitigate these Apple open changes, read this blog post: iOS 15: How to Prepare for Apple’s Changes.

mceclip3.png

Klaviyo offers smart test size recommendations based on the size of your recipient list. To change the test size, use the slider bar. 

If the size is less than 100%, choose your test duration. 

For the example below, 20% of the campaign’s recipients will get variation A, and another 20% will get variation B. Depending on how many of these recipients open or click the email, a winner will be chosen after six hours, and the rest of the recipients will receive the winning variation. 

An example of an A/B test that looks at how 40% of the campaign's recipients respond to the variations over the course of 6 hours.

After you review or adjust the test’s settings, click Continue to Review

What to A/B Test 

Content

Now that you know how to perform an A/B test, it’s time to decide what to test. Focus on only one aspect at a time and follow these best practices for A/B testing

Examples of what to A/B test include: 

  • Subject lines: test different phrases and whether customers like short or long subjects.
  • Discounts (10%, 20%, no offer): see how discounts may/may not drive increased clicks.
  • Email design: figure out whether plain-text or graphically rich HTML emails perform better in terms of click rates and conversions.
  • URL parameters: test how different links drive customers to the same content. You can accomplish this in Klaviyo by differentiating your links with different parameters.

A/B Test Exceeded Account Sending Limits

Klaviyo will automatically cancel A/B test campaigns when the testing pool exceeds an account's monthly sending limits. 

If you receive an in-app notification that reads: "Your A/B test campaign CAMPAIGN_NAME was automatically cancelled for exceeding account sending limits," you will need to upgrade to a higher plan in order to resend the campaign.

To view your account's monthly sending limit, head to your account dropdown in the upper right-hand corner of your screen and select Account. Here, you’ll see an Account Overview where you can upgrade your plan. 

The Account Overview page where you can change your billing plan.

Once you have done so, navigate to your Campaigns tab, find the cancelled campaign, and click Clone. From there, you can configure and send your new campaign.

Review A/B Test Results

Navigate to the campaign you are testing.

Select the A/B Test Results tab of the main campaign report.For more information, check out our Review Your Email A/B Test Results article.

Additional Resources

Read other articles on A/B testing: 

x
Was this article helpful?
326 out of 404 found this helpful