You will learn
Learn how to set up and run a campaign A/B test, how to read the results of an A/B test, and a few use cases for A/B testing campaigns. Klaviyo's A/B testing feature for campaigns allows you to easily test different subject lines and content to refine your messaging and optimize your send times.
How to A/B test campaign emails
- Navigate to Campaigns > Create Campaign > Email
- Name your campaign
- Select the lists or segments you want to send to
- Click Continue to Content
- Input the subject line and, if desired, edit the preview text, sender name, and sender email address.
- Create the first version of your email, adding your text, images, and links
- Click Save Content
- At the bottom of the page, click Create A/B Test
This will automatically create a second, identical variation of your campaign and bring you to an A/B testing page.
On this page, edit one of the variations for your A/B test. It’s important for A/B tests to compare one factor at a time, so if you edit the subject line, do not change anything else.
Clone a variation
If desired, you can add more variations by clicking the Clone button, although we recommend sticking to just two variations.
Test send times
You can also test send times to determine when your audience wants to hear from you. Note that you can only select open rate as the winning metric when testing send times.
To do so, click Switch to testing send times at the top of the page.
In this case, keep the content of both variations exactly the same. Then, choose the times you want to test.
If you don't know where to start, we recommend trying an evening time. Alternatively, if you qualify, you can use Smart Send Time instead.
Configure A/B test settings
After creating your content, decide the size of your testing pool and the length of your testing period.
First, choose the winning metric, which is the main metric you want to see improve. The default is open rate, but you can switch it to click rate.
- Open rate is best if you are testing differences in:
- Subject line
- Preview text
- Sender name/email address
- Click rate is best if you are testing differences in the body of the email
With the release of iOS15, macOS Monterey, iPadOS 15, and WatchOS 8, Apple Mail Privacy Protection (MPP) changed the way that we receive open rate data on your emails by prefetching our tracking pixel. With this change, it’s important to understand that open rates will be inflated.
With regard to A/B testing, we anticipate that our tools should account for these inflated open rates; however, you may need a higher threshold to reach statistical significance. If you regularly conduct A/B testing AND have greater than ~45% of opens on Apple Mail, we suggest creating a custom report that includes an MPP property. You can also identify these opens in your individual subscriber segments.
For complete information on MPP opens, visit our iOS 15: How to prepare for Apple’s changes guide.
Klaviyo offers smart test size recommendations based on the size of your recipient list. To change the test size, use the slider bar.
If the size is less than 100%, choose your test duration.
For the example below, 20% of the campaign’s recipients will get variation A, and another 20% will get variation B. Depending on how many of these recipients open or click the email, a winner will be chosen after six hours, and the rest of the recipients will receive the winning variation.
After you review or adjust the test’s settings, click Continue to Review.
What to A/B test
Now that you know how to perform an A/B test, it’s time to decide what to test. Focus on only one aspect at a time and follow these best practices for A/B testing.
Examples of what to A/B test include:
- Subject lines: test different phrases and whether customers like short or long subjects.
- Discounts (10%, 20%, no offer): see how discounts may/may not drive increased clicks.
- Email design: figure out whether plain-text or graphically rich HTML emails perform better in terms of click rates and conversions.
- URL parameters: test how different links drive customers to the same content. You can accomplish this in Klaviyo by differentiating your links with different parameters.
A/B test exceeded account sending limits
Klaviyo will automatically cancel A/B test campaigns when the testing pool exceeds an account's monthly sending limits.
If you receive an in-app notification that reads: "Your A/B test campaign CAMPAIGN_NAME was automatically cancelled for exceeding account sending limits," you will need to upgrade to a higher plan in order to resend the campaign. We also suggest turning on the auto-upgrade billing option to avoid hitting this threshold in the future. With auto-upgrade, we will move you up automatically to the next plan level, should you exceed your current plan limits.
To view your account's monthly sending limit, head to your account dropdown in the upper right-hand corner of your screen and select Account. Here, you’ll see an Account Overview where you can upgrade your plan.
Once you have done so, navigate to your Campaigns tab, find the cancelled campaign, and click Clone. From there, you can configure and send your new campaign.
Review A/B test results
Navigate to the campaign you are testing.
Select the A/B Test Results tab of the main campaign report.For more information, check out How to review your email A/B test results for campaigns.