How to A/B test campaign emails

read
Last updated at:

You will learn

Learn how to set up and run a campaign A/B test, how to read the results of an A/B test, and a few use cases for A/B testing campaigns. Klaviyo's A/B testing feature for campaigns allows you to easily test different subject lines, message content, and send times so you can better understand what works best for your audience. 

How to A/B test campaign emails

  1. Navigate to Campaigns > Create Campaign > Email.
  2. Name your campaign.
  3. Select the lists or segments you want to send to.
  4. Click Continue to Content.
  5. Input the subject line and, if desired, edit the preview text, sender name, and sender email address. 
  6. Create the first version of your email, adding your copy, images, and links.
  7. Click Save Content.
  8. At the bottom of the page, click Create A/B Test.
    The Create A/B test button is located at the bottom of the email preview screen.

This will automatically create a second, identical variation of your campaign and bring you to the Campaign A/B Test page.

Both variations of an A/B test.

Test content

On this page, edit one of the variations for your A/B test. It’s important for A/B tests to compare one factor at a time, so if you edit the subject line, do not change anything else. 

Clone a variation

If desired, you can add more variations by clicking the Clone button, although we recommend sticking to just 2 variations. 

The Clone option, wbich is available for both variations.

Test send times

You can also test send times to determine when your audience wants to hear from you. To do so, click Switch to testing send times at the top of the page.

The testing send times option is located in the upper right of the A/B test Settings page.

When testing send times, keep the content and subject lines of both variations exactly the same. 

If you don't know where to start, we recommend trying an evening time. Alternatively, if you qualify, you can use Smart Send Time instead. 

Configure A/B test settings

After creating your content, decide the size of your testing pool and the length of your testing period.

First, choose the winning metric, which is the main metric you want to see improve. Choose from: 

  • Open rate
    Recommended when testing subject line, preview text, or sender name/email address.
  • Click rate
    Recommended when testing email content, like button size or color.
  • Placed order rate
    Only available for accounts with a Placed Order metric; recommended when testing email content, like whether displaying best sellers or new items leads to more conversions. 

An A/B tests Winning Metric options, which include opens, clicks, or placed orders

With the release of iOS15, macOS Monterey, iPadOS 15, and WatchOS 8, Apple Mail Privacy Protection (MPP) changed the way that we receive open rate data on your emails by prefetching our tracking pixel. With this change, it’s important to understand that open rates will be inflated.

With regard to A/B testing, we anticipate that our tools should account for these inflated open rates; however, you may need a higher threshold to reach statistical significance. If you regularly conduct A/B testing AND have greater than ~45% of opens on Apple Mail, we suggest creating a custom report that includes an MPP property. You can also identify these opens in your individual subscriber segments.

For complete information on MPP opens, visit our iOS 15: Everything you need to know about Apple’s changes guide.

Klaviyo recommends a test size based on your list size and a test duration based on the A/B test metric (i.e., opens, clicks, or placed orders). Use the slider bar to change the test size if desired. If the test size is less than 100%, you also have the option to adjust your test duration. 

For the example below, 20% of the campaign’s recipients will get variation A, and another 20% will get variation B. Depending on how many of these recipients open, click, or place an order (depending on the winning metric you selected), a winner will be chosen after 6 hours, and the rest of the recipients will receive the winning variation. 

An example of an A/B test that looks at how 40% of the campaign's recipients respond to the variations over the course of 6 hours.

After you review and adjust the test’s settings, click Continue to Review

What to A/B test 

You should only test one variable at a time. Testing multiple variables simultaneously can make it difficult (or impossible) to identify which variables contribute to improved performance. Not sure what to test? Use the ideas below as a starting point. 

Tests to improve your placed order rates

To see what drives your recipients to place orders, try testing any of the following in your messages: 

  • Social proof (e.g., including positive reviews or customer social media posts in your messages)
  • Send time, day of week, or weekends versus weekdays
  • Call to action (CTA) placement (e.g., near the top of your email or lower down)

Tests to improve your click rates

To see what drives your recipients to click on links in your messages, try testing any of the following: 

  • CTA appearance (e.g., the font, color, or size of your button or text)
  • CTA text (e.g., "Shop now" versus "Check it out!")
  • Template organization (e.g., how you arrange the images and text in your message)
  • Images versus GIFs

Tests to improve your open rates 

To identify what prompts your subscribers to open your emails, try testing any of the following: 

  • Subject line tests: 
    • Statements versus questions (e.g., “You’ve gotta see our latest drop” versus “Want to see our latest drop?”)
    • Subject line length
    • Subject line personalization (e.g., including the recipient’s first name in the subject line using the {{ first_name|default:’Friend’ }} tag)
    • Emojis in subject lines
  • Preview text (e.g., including versus not including preview text)
  • From name (e.g., your company name compared to a more personal From name, like “Elise at Klaviyo”)
  • Send time, day of week, or weekends versus weekdays

Review A/B test results and select a winner (optional)

Navigate to the campaign you are testing to view its progress and see which version has performed better. If you'd like to end an ongoing A/B test:

  1. Navigate to the A/B Test Results tab of an ongoing campaign.
  2. Click the additional options menu next to the version you'd like to choose.
  3. Click Choose as Winner.

When you manually select a winner, anyone who hasn't yet received the campaign will receive the winning variation immediately. For more information, check out How to review your email A/B test results for campaigns.

Additional resources

x
Was this article helpful?
487 out of 613 found this helpful