You will learn
Learn how to set up and run a campaign A/B test, how to read the results of an A/B test, and a few use cases for A/B testing campaigns. Klaviyo's A/B testing feature for campaigns allows you to easily test different subject lines, message content, and send times so you can better understand what works best for your audience.
How to A/B test campaign emails
- Navigate to Campaigns > Create Campaign > Email.
- Name your campaign.
- Select the lists or segments you want to send to.
- Click Continue to Content.
- Input the subject line and, if desired, edit the preview text, sender name, and sender email address.
- Create the first version of your email, adding your copy, images, and links.
- Click Save Content.
- At the bottom of the page, click Create A/B Test. This will automatically create a second, identical variation of your campaign and bring you to the Campaign A/B Test page.
- In each Variation Name field, add a descriptive variation name that references what you're A/B testing.
When you create an A/B test for a campaign with a template originally built in Klaviyo’s classic template editor, both variations of the A/B test will use Klaviyo’s new template editor. Learn more about when templates automatically switch to the new editor.
Test content
On this page, edit one of the variations for your A/B test. It’s important for A/B tests to compare one factor at a time, so if you edit the subject line, do not change anything else.
Clone a variation
If desired, you can add more variations by clicking the Clone button, although we recommend sticking to just 2 variations.
When you clone an A/B test variation originally built in Klaviyo’s classic template editor, the new, cloned version of the template will use Klaviyo’s new template editor. Learn more about when templates automatically switch to the new editor.
Test send times
You can also test send times to determine when your audience wants to hear from you. To do so, click Switch to testing send times at the top of the page.
When testing send times, keep the content and subject lines of both variations exactly the same.
If you don't know where to start, we recommend trying an evening time. Alternatively, if you qualify, you can use Smart Send Time instead.
Configure A/B test settings
After creating your content, decide the size of your testing pool and the length of your testing period.
First, choose the winning metric, which is the main metric you want to see improve. Choose from:
- Open rate
Recommended when testing subject line, preview text, or sender name/email address. - Click rate
Recommended when testing email content, like button size or color. - Placed order rate
Only available for accounts with a Placed Order metric; recommended when testing email content, like whether displaying best sellers or new items leads to more conversions.
With the release of iOS15, macOS Monterey, iPadOS 15, and WatchOS 8, Apple Mail Privacy Protection (MPP) changed the way that we receive open rate data on your emails by prefetching our tracking pixel. With this change, it’s important to understand that open rates will be inflated.
With regard to A/B testing, we anticipate that our tools should account for these inflated open rates; however, you may need a higher threshold to reach statistical significance. If you regularly conduct A/B testing AND have greater than ~45% of opens on Apple Mail, we suggest creating a custom report that includes an MPP property. You can also identify these opens in your individual subscriber segments.
For complete information on MPP opens, visit our iOS 15: Everything you need to know about Apple’s changes guide.
Klaviyo recommends a test size based on your list size and a test duration based on the A/B test metric (i.e., opens, clicks, or placed orders). Use the slider bar to change the test size if desired. If the test size is less than 100%, you also have the option to adjust your test duration.
For the example below, 20% of the campaign’s recipients will get variation A, and another 20% will get variation B. Depending on how many of these recipients open, click, or place an order (depending on the winning metric you selected), a winner will be chosen after 6 hours, and the rest of the recipients will receive the winning variation.
After you review and adjust the test’s settings, click Continue to Review.
If you are A/B testing content and would like to send your message based on each recipient’s timezone, set your test size to 100%. When scheduling your campaign, you’ll have the option to choose Recipient’s Local Timezone as the timezone for your send.
What to A/B test
You should only test one variable at a time. Testing multiple variables simultaneously can make it difficult (or impossible) to identify which variables contribute to improved performance. Not sure what to test? Use the ideas below as a starting point.
Tests to improve your placed order rates
To see what drives your recipients to place orders, try testing any of the following in your messages:
- Social proof (e.g., including positive reviews or customer social media posts in your messages)
- Send time, day of week, or weekends versus weekdays
- Call to action (CTA) placement (e.g., near the top of your email or lower down)
Tests to improve your click rates
To see what drives your recipients to click on links in your messages, try testing any of the following:
- CTA appearance (e.g., the font, color, or size of your button or text)
- CTA text (e.g., "Shop now" versus "Check it out!")
- Template organization (e.g., how you arrange the images and text in your message)
- Images versus GIFs
Tests to improve your open rates
To identify what prompts your subscribers to open your emails, try testing any of the following:
- Subject line tests:
- Statements versus questions (e.g., “You’ve gotta see our latest drop” versus “Want to see our latest drop?”)
- Subject line length
- Subject line personalization (e.g., including the recipient’s first name in the subject line using the {{ first_name|default:’Friend’ }} tag)
- Emojis in subject lines
- Preview text (e.g., including versus not including preview text)
- From name (e.g., your company name compared to a more personal From name, like “Elise at Klaviyo”)
- Send time, day of week, or weekends versus weekdays
Review A/B test results and select a winner (optional)
Navigate to the campaign you are testing to view its progress and see which version has performed better. If you'd like to end an ongoing A/B test:
- Navigate to the A/B Test Results tab of an ongoing campaign.
- Click the additional options menu next to the version you'd like to choose.
- Click Choose as Winner.
When you manually select a winner, anyone who hasn't yet received the campaign will receive the winning variation immediately. If your A/B test size was 100% of recipients, we will continue collecting data throughout the conversion window for the message, so the winner may change as new data comes in. For more information, check out How to review your email A/B test results for campaigns.