How to A/B test campaign emails

Last updated at:

You will learn

Learn how to set up and run a campaign A/B test, how to read the results of an A/B test, and a few use cases for A/B testing campaigns. Klaviyo's A/B testing feature for campaigns allows you to easily test different subject lines and content to refine your messaging and optimize your send times. 

How to A/B test campaign emails

  1. Navigate to Campaigns > Create Campaign > Email
  2. Name your campaign
  3. Select the lists or segments you want to send to
  4. Click Continue to Content
  5. Input the subject line and, if desired, edit the preview text, sender name, and sender email address. Step one of the campaign creation wizard
  6. Create the first version of your email, adding your text, images, and links
  7. Click Save Content
  8. At the bottom of the page, click Create A/B TestThe Create A/B test button is located at the bottom of the email preview screen.

This will automatically create a second, identical variation of your campaign and bring you to an A/B testing page.

The A/B Test Settings page.

Test content

On this page, edit one of the variations for your A/B test. It’s important for A/B tests to compare one factor at a time, so if you edit the subject line, do not change anything else. 

Clone a variation

If desired, you can add more variations by clicking the Clone button, although we recommend sticking to just two variations. 

The Clone option, wbich is available for both variations.

Test send times

You can also test send times to determine when your audience wants to hear from you. Note that you can only select open rate as the winning metric when testing send times.

To do so, click Switch to testing send times at the top of the page.

In this case, keep the content of both variations exactly the same. Then, choose the times you want to test. 

The testing send times option is located in the upper right of the A/B test Settings page.

If you don't know where to start, we recommend trying an evening time. Alternatively, if you qualify, you can use Smart Send Time instead. 

Configure A/B test settings

After creating your content, decide the size of your testing pool and the length of your testing period.

First, choose the winning metric, which is the main metric you want to see improve. The default is open rate, but you can switch it to click rate.

  • Open rate is best if you are testing differences in:
    • Subject line
    • Preview text
    • Sender name/email address
  • Click rate is best if you are testing differences in the body of the email

With the release of iOS15, macOS Monterey, iPadOS 15, and WatchOS 8, Apple Mail Privacy Protection (MPP) changed the way that we receive open rate data on your emails by prefetching our tracking pixel. With this change, it’s important to understand that open rates will be inflated.

With regard to A/B testing, we anticipate that our tools should account for these inflated open rates; however, you may need a higher threshold to reach statistical significance. If you regularly conduct A/B testing AND have greater than ~45% of opens on Apple Mail, we suggest creating a custom report that includes an MPP property. You can also identify these opens in your individual subscriber segments.

For complete information on MPP opens, visit our iOS 15: How to prepare for Apple’s changes guide.


Klaviyo offers smart test size recommendations based on the size of your recipient list. To change the test size, use the slider bar. 

If the size is less than 100%, choose your test duration. 

For the example below, 20% of the campaign’s recipients will get variation A, and another 20% will get variation B. Depending on how many of these recipients open or click the email, a winner will be chosen after six hours, and the rest of the recipients will receive the winning variation. 

An example of an A/B test that looks at how 40% of the campaign's recipients respond to the variations over the course of 6 hours.

After you review or adjust the test’s settings, click Continue to Review

What to A/B test 


Now that you know how to perform an A/B test, it’s time to decide what to test. Focus on only one aspect at a time and follow these best practices for A/B testing

Examples of what to A/B test include: 

  • Subject lines: test different phrases and whether customers like short or long subjects.
  • Discounts (10%, 20%, no offer): see how discounts may/may not drive increased clicks.
  • Email design: figure out whether plain-text or graphically rich HTML emails perform better in terms of click rates and conversions.
  • URL parameters: test how different links drive customers to the same content. You can accomplish this in Klaviyo by differentiating your links with different parameters.

A/B test exceeded account sending limits

Klaviyo will automatically cancel A/B test campaigns when the testing pool exceeds an account's monthly sending limits. 

If you receive an in-app notification that reads: "Your A/B test campaign CAMPAIGN_NAME was automatically cancelled for exceeding account sending limits," you will need to upgrade to a higher plan in order to resend the campaign. We also suggest turning on the auto-upgrade billing option to avoid hitting this threshold in the future. With auto-upgrade, we will move you up automatically to the next plan level, should you exceed your current plan limits.

To view your account's monthly sending limit, head to your account dropdown in the upper right-hand corner of your screen and select Account. Here, you’ll see an Account Overview where you can upgrade your plan. 

The Account Overview page where you can change your billing plan.

Once you have done so, navigate to your Campaigns tab, find the cancelled campaign, and click Clone. From there, you can configure and send your new campaign.

Review A/B test results

Navigate to the campaign you are testing.

Select the A/B Test Results tab of the main campaign report.For more information, check out How to review your email A/B test results for campaigns.

Additional resources

Was this article helpful?
434 out of 543 found this helpful