You will learn
Learn how to use Klaviyo’s updated A/B testing experience to test individual emails in a flow, including things like subject line, discounts, and email content, etc. You can even use A/B tests to determine if plain-text emails perform better than image-heavy HTML emails.
All new tests you create will automatically be using the updated A/B testing experience. However, you may want to migrate existing tests to use the updated experience as well. If you have not done so already, please follow this guide on How to migrate existing tests to the updated A/B testing experience.
About flows and A/B testing
Flow email A/B testing includes the processes of:
- Creating a single flow email and scheduling this email to send after a certain number of hours or days, just as you normally would.
- Configuring two or more email variations for this single flow email — adjusting the subject line or core email content for each variation.
- Selecting the weight of each variation to establish the percentage of recipients that will receive each email.
- Having a winning variation chosen automatically after statistical significance is reached, or manually choosing a winner based on the highest open rate or highest click rate amongst variations.
The updated A/B testing experience offers improved performance analysis, efficiency, and visibility, including the ability to automatically choose test winners. You can more efficiently run multiple tests for the same message at different times, and you will have the ability to run multiple tests for the same message at different times.
Perform a flow email A/B test
Create a new test
- To set up A/B testing for an individual email in any flow, click on the email within the flow builder.
- Click Create A/B Test in the upper right-hand corner of the sidebar.
When you create an A/B test for an existing flow email, two new copies of the email are created. If the flow email is live and has already been sending, when the A/B test is published the analytics for each of these copies are tracked separately. We do not continue tracking analytics for the original email once you have added a variation.
After creating a new test, you will be taken directly to the A/B Test tab of the Manage Content window. Every test will automatically include 2 variations. When you click on a variation, you can edit the variation name, subject line, preview text, sender name, and sender email address.
- To edit the content of the email, click the Edit Email button above the email preview on the right.
Add a new variation
If you’d like to add another variation, you must clone one of the existing 2 variations.
- Click the clone icon to the right of the variation you want to base your new variation on.
- This will create a new variation which you can modify to be distinct from the original 2 variations.
Delete a variation
An A/B test must have at least 2 variations, so a variation can only be deleted when there are 3 or more. Additionally, variations can only be deleted while a test is in Draft status. Deleting variations is disabled for live A/B tests.
- To delete a variation, click on the trashcan icon to the right of the variation you want to delete.
Choose the weight of each variation
At the bottom of the A/B Test tab, you will see the Test Settings section where you can edit the weight of each variation – this will establish the percentage of recipients that will receive each variation.
It's generally recommended to weigh each variation equally, but you can change the percentages to any amount you’d like.
You can also select the option to automatically choose distribution. This will set the distribution to be equal for each variation at first. Over time, if one of the variations performs better than the others, Klaviyo will automatically adjust the distribution to direct more recipients to the better performing variation while the test is still running.
Configure winner selection
Tests will default to automatically choose a winner based on click rate, but this can be changed.
- In the Test Settings section at the bottom of the A/B Test tab, you can also configure winner selection by clicking on the Configure link under Winner Selection.
- A modal will appear in which you can change the Winning Metric to Open Rate instead.
Using Click Rate is recommended since many factors such as Apple Mail Privacy Protection (MPP) can cause inflated open rates. For complete information on MPP opens, visit our iOS 15: How to Prepare for Apple’s Changes guide.
- In the Automatic Winner Selection section, there are 2 options:
- End test if the winning metric is determined to win with high statistical certainty (statistical significance): this option is checked by default. Uncheck the box next to this option if you’d like to manually end the test instead or choose the option to end the test after a specific date.
- Set a date limit: this option allows you to select a specific date to end the test automatically.
- If you select both options, the test will end if statistical significance is reached or if the specified date is reached depending on which occurs first.
- Click the Save Settings button to confirm your configuration.
Publish a test
- Once you have your variations configured, you are ready to publish your test. From the A/B Test tab, click the Publish Test button in the upper right.
- A modal will appear asking you to confirm the publication. Click the Publish Test button to confirm.
Delete a test
- If you wish to delete a test, from the A/B Test tab, click the arrow next to the Publish Test button and select Delete.
View A/B test results
- You can view the results of a test while it is running or after completion by clicking the email within the flow builder and clicking the View A/B Test button.
This will take you to the A/B Test Results page where you can view the following metrics:
How many people received the email
What percentage/number of people opened the email
What percentage/number of people clicked through the email
- Conversion Metric
Total recipients who triggered the conversion metric the selected conversion metric for the email such as Placed Order
Manually choose a winner
- From the A/B Test Results page you can manually choose a winning variation for the test by clicking the Choose Winner button in the top right corner of the page.
- A modal will appear asking you to choose a winner from each of your existing variations. Select the variation you’d like to win from the dropdown then click the Choose Winner button.
Changing a message’s status from live to draft status will end the test and automatically choose a winner based on the winning metric – the first variation will be chosen if there is a tie.
Learn more about A/B testing in flows: