How to A/B test campaign emails

read
Last updated at:

You will learn

Learn how to set up and run a campaign A/B test, how to read the results of an A/B test, and a few use cases for A/B testing campaigns. Klaviyo's A/B testing feature for campaigns allows you to easily test different subject lines, message content, and send times so you can better understand what works best for your audience. 

How to A/B test campaign emails

  1. Navigate to Campaigns > Create Campaign > Email
  2. Name your campaign
  3. Select the lists or segments you want to send to
  4. Click Continue to Content
  5. Input the subject line and, if desired, edit the preview text, sender name, and sender email address 
  6. Create the first version of your email, adding your copy, images, and links
  7. Click Save Content
  8. At the bottom of the page, click Create A/B Test
    The Create A/B test button is located at the bottom of the email preview screen.

This will automatically create a second, identical variation of your campaign and bring you to the Campaign A/B Test page.

Both variations of an A/B test.

Test content

On this page, edit one of the variations for your A/B test. It’s important for A/B tests to compare one factor at a time, so if you edit the subject line, do not change anything else. 

Clone a variation

If desired, you can add more variations by clicking the Clone button, although we recommend sticking to just 2 variations. 

The Clone option, wbich is available for both variations.

Test send times

You can also test send times to determine when your audience wants to hear from you. To do so, click Switch to testing send times at the top of the page.

The testing send times option is located in the upper right of the A/B test Settings page.

When testing send times, keep the content and subject lines of both variations exactly the same. 

If you don't know where to start, we recommend trying an evening time. Alternatively, if you qualify, you can use Smart Send Time instead. 

Configure A/B test settings

After creating your content, decide the size of your testing pool and the length of your testing period.

First, choose the winning metric, which is the main metric you want to see improve. Choose from: 

  • Open rate
    Recommended when testing subject line, preview text, or sender name/email address
  • Click rate
    Recommended when testing email content, like button size or color
  • Placed order rate
    Only available for accounts with a Placed Order metric; recommended when testing email content, like whether displaying best sellers or new items leads to more conversions 

The option to choose Placed order rate as a winning metric for an A/B test is currently available to a limited number of Klaviyo users, and will be available to everyone in the coming weeks. 

An A/B tests Winning Metric options, which include opens, clicks, or placed orders

With the release of iOS15, macOS Monterey, iPadOS 15, and WatchOS 8, Apple Mail Privacy Protection (MPP) changed the way that we receive open rate data on your emails by prefetching our tracking pixel. With this change, it’s important to understand that open rates will be inflated.

With regard to A/B testing, we anticipate that our tools should account for these inflated open rates; however, you may need a higher threshold to reach statistical significance. If you regularly conduct A/B testing AND have greater than ~45% of opens on Apple Mail, we suggest creating a custom report that includes an MPP property. You can also identify these opens in your individual subscriber segments.

For complete information on MPP opens, visit our iOS 15: Everything you need to know about Apple’s changes guide.

Klaviyo recommends a test size based on your list size and a test duration based on the A/B test metric (i.e., opens, clicks, or placed orders). Use the slider bar to change the test size if desired. If the test size is less than 100%, you also have the option to adjust your test duration. 

For the example below, 20% of the campaign’s recipients will get variation A, and another 20% will get variation B. Depending on how many of these recipients open, click, or place an order (depending on the winning metric you selected), a winner will be chosen after 6 hours, and the rest of the recipients will receive the winning variation. 

An example of an A/B test that looks at how 40% of the campaign's recipients respond to the variations over the course of 6 hours.

After you review and adjust the test’s settings, click Continue to Review

What to A/B test 

Content

Now that you know how to perform an A/B test, it’s time to decide what to test. Focus on only one aspect at a time and follow these best practices for A/B testing

Examples of what to A/B test include: 

  • Subject lines: test different phrases, whether customers like short or long subjects, different tones, or emojis
  • Discounts (10%, 20%, no offer): see how discounts may/may not drive increased clicks or placed orders
  • Email design: determine whether plain-text or graphically rich HTML emails perform better in terms of click rates or conversions
  • Call to action (CTA) design: test different CTA styles, like button colors, sizes, or placement to see which leads to more clicks or conversions

A/B test exceeded account sending limits

Klaviyo will automatically cancel A/B test campaigns when the testing pool exceeds an account's monthly sending limits. If a campaign is canceled for this reason, it will not automatically continue sending if you later upgrade your plan. 

If you receive an in-app notification that reads: "Your A/B test campaign CAMPAIGN_NAME was automatically canceled for exceeding account sending limits," you will need to upgrade to a higher plan in order to resend the campaign. We also suggest turning on the auto-upgrade billing option to avoid hitting this threshold in the future. With auto-upgrade, we will move you up automatically to the next plan level, should you exceed your current plan limits.

To view your account's monthly sending limit, head to your account menu and select Account. Here, you’ll see an Account Overview where you can upgrade your plan. 

Once you have done so, to continue sending the campaign: 

  1. Navigate to the Lists & Segments tab 
  2. Create a segment with the following definition:
    What someone has done > Received Email > at least once over all time > where > Campaign Name equals [your campaign name]
  3. Navigate to your Campaigns tab and locate the canceled campaign
  4. Click Clone
  5. Exclude the segment you created in step 2 from the cloned campaign
  6. Send the campaign

By following the steps above, no one who received your message before you ran out of funds will receive the second, re-sent campaign. 

Review A/B test results and select a winner (optional)

Navigate to the campaign you are testing to view its progress and see which version has performed better. If you'd like to end an ongoing A/B test:

  1. Navigate to the A/B Test Results tab of an ongoing campaign
  2. Click the additional options menu next to the version you'd like to choose
  3. Click Choose as Winner

When you manually select a winner, anyone who hasn't yet received the campaign will receive the winning variation immediately. For more information, check out How to review your email A/B test results for campaigns.

Additional resources

x
Was this article helpful?
457 out of 578 found this helpful