How to A/B test an email campaign

You will learn

Learn how to set up and run an A/B test for a campaign, how to read the test results, and a few use cases for A/B testing campaigns. Klaviyo's A/B testing feature for campaigns allows you to easily test different subject lines, message content, and send times so you can better understand what works best for your audience.

Create an A/B test for campaign emails

1. Navigate to Campaigns > Create campaign.
2. In the Create campaign window, name your campaign.
3. Choose Single channel, then Email, and click Continue.
4. Select the lists or segments you want to send to.
5. Click Next.
6. Input the subject line and, if desired, edit the preview text, sender name, and sender email address.
7. Create the first version of your email, adding your copy, images, and links.
8. To create the test manually, click Create A/B test above the subject line field. To generate a subject-line variation with AI instead, skip to Test subject lines with AI below before creating the test.
   • This automatically creates a second, identical variation of your campaign and opens the variations for editing.

9. In each Variation name field, add a descriptive variation name that references what you're A/B testing, i.e., “Summer launch email - white buttons.”
   

Configure your test variations

Decide which element of your campaign to test, then configure your variations using one of the options below.

Test subject lines with AI

Instead of creating a test manually, you can also automate creation using AI. This feature is for testing subject lines in particular.

1. Start with an email campaign that has only 1 variation. If you are following the steps above, stop before clicking Create A/B test.
2. Enter your subject line, choose a template, and edit the email content if you have not already done so.
3. Underneath the Subject line field, choose one of the options in the Generate a variation to A/B test section.
   

After you choose an option, AI creates a test with 2 variations: variation A keeps your original subject line, and variation B uses a subject line generated from your selected prompt.

Choose one of the following options. The generated variation depends on your original subject line:

• Personalization vs no personalization
  Variation B will include the customer's first name in the subject line or remove their name if it was originally included.
• Short vs long
  Variation B will either have a shorter or longer subject line depending on the length of your original subject.
• Question vs statement
  Variation B will convert your original subject line into a question or into a statement if it was originally a question.
• Emoji vs no emoji
  Variation B will include a relevant emoji in the subject line or remove any emojis if they were originally included.

Read the rest of this article for steps on how to further edit and send your A/B test once it's been created.

Test content

You can test message content to determine what your audience wants to see. In A/B test setup, select Test content.

Edit one of the variations for your A/B test. It’s important for A/B tests to compare one factor at a time, so if you edit the subject line, do not change anything else. If desired, you can add more variations by clicking the Clone icon on the card for a specific variation, but we recommend only 2 variations.

Test send times

You can also test send times to determine when your audience wants to hear from you. In A/B test setup, select Test send times.

When testing send times, keep the content and subject lines of both variations exactly the same.

For each variation, choose a Send date and Send time. These times use your account’s time zone. Send-time tests send to 100% of the campaign’s recipients, so there is no remaining audience to receive a winning variation after the test.

If you don't know where to start, we recommend trying an evening time. Alternatively, if you qualify, you can use smart send time instead.

Select a test strategy

If your account qualifies, decide on your testing strategy from the following options:

• Winning variation (Standard A/B test)
• Personalized variations for each recipient

The difference between winning variation and personalized variations

When a content test uses less than 100% of the campaign’s recipients, both strategies send variations to a test group and measure their success using your chosen metric, such as open rate. After the test period, they send to the remaining recipients differently:

• Winning variation - the default option for A/B tests, which will determine 1 winning variation and send it to the rest of the campaign's recipients after the test period is over.
• Personalized variations - uses AI to search for patterns amongst the test recipients who interact with each variation. After the testing period is over, Klaviyo will predict which variation will perform better for each recipient and send the rest of the recipients their preferred variation.

Personalize variations for each recipient

If you’d like to personalize which variation each recipient receives on an individual basis, select Personalized variations for each recipient.

Klaviyo will use information about each profile to determine which variation is most likely to succeed in converting that profile. This profile information includes, but is not limited to:

• Historical engagement rates
• Customer lifetime value (CLV)
• Location

For example, if you have chosen open rate as your winning metric and profiles in the test group with a CLV of 100 or more open variation A more frequently, but profiles with a CLV of less than 100 open variation B more frequently, the rest of the campaign's recipients will receive emails based on which variation they are most likely to open according to their CLV. This is a simplified example, as Klaviyo will use many data points to determine personalized variations.

If there is not enough engagement to personalize variations, Klaviyo sends the winning variation to the remaining recipients instead.

Configure A/B test settings

After configuring your variations, choose the test settings. For content tests, you can adjust the test size and, when the test size is below 100%, the test duration.

First, choose the winning metric, which is the main metric you want to see improve. Choose from:

• Open rate
  Recommended when testing subject line, preview text, or sender name and email address.
• Click rate
  Recommended when testing email content, like button size or color.
• Placed order rate
  Only available for accounts with a Placed Order metric and not available for personalized variations. This is recommended when testing email content, like whether displaying best sellers or new items leads to more conversions.

For content tests, Klaviyo recommends a test size based on your audience size and a test duration based on the winning metric (opens, clicks, or placed orders). Use the slider to change the test size if desired. If the test size is less than 100%, you can also adjust the test duration. Send-time tests have a fixed test size of 100%.

For a standard content test with a 40% test size and a 6-hour test duration, 20% of the campaign’s recipients will get variation A, and another 20% will get variation B. Depending on how many of these recipients open, click, or place an order (depending on the winning metric you selected), a winner will be chosen after 6 hours, and the rest of the recipients will receive the winning variation.

Schedule or send your test

Follow these steps to send or schedule your test:

1. If you want a content test to send in each recipient’s local time zone, set Test size to 100% before continuing.

2. Click Next to review your campaign.
3. Review the campaign details and resolve any errors, then click Schedule or send.
4. For a content test, choose Send now or Schedule. If scheduling, choose the date, time, and time zone. To send in each recipient’s local time zone, select Recipient's Local Time Zone. For a send-time test, review the dates and times already set for each variation.

5. Click Send now or Schedule message, as appropriate, to confirm.

What to A/B test

You should only test one variable at a time. Testing multiple variables simultaneously can make it difficult (or impossible) to identify which variables contribute to improved performance. Not sure what to test? Click on the sections below to view starting points.

Review A/B test results and select a winner (optional)

Open the campaign to compare each variation’s performance and review its win probability. To end a running content test early and choose a winner manually, the test size must be less than 100% so that some recipients remain to receive the winner.

Choosing a winner ends the test and starts sending that variation to the remaining recipients immediately.

1. Navigate to the A/B test results tab of the campaign.
2. Click the additional options menu next to the version you'd like to choose.
3. Click Choose as winner.
4. In the confirmation window, click Choose winner.

If your A/B test size was 100% of recipients, there is no later send of the winning variation. The reported winner may change as new engagement data comes in. For more information, check out How to review your email A/B test results for campaigns.

Additional resources