How to A/B test a flow email using the updated A/B testing experience

read
Last updated at:

You will learn

Learn how to use Klaviyo’s updated A/B testing experience to test individual emails in a flow, including things like subject line, discounts, and email content, etc. You can even use A/B tests to determine if plain-text emails perform better than image-heavy HTML emails.

All new tests you create will automatically be using the updated A/B testing experience. However, you may want to migrate existing tests to use the updated experience as well. If you have not done so already, please follow this guide on How to migrate existing tests to the updated A/B testing experience.

About flows and A/B testing

Flow email A/B testing includes the processes of:

  1. Creating a single flow email and scheduling this email to send after a certain number of hours or days, just as you normally would.
  2. Configuring two or more email variations for this single flow email — adjusting the subject line or core email content for each variation.
  3. Selecting the weight of each variation to establish the percentage of recipients that will receive each email.
  4. Having a winning variation chosen automatically after statistical significance is reached, or manually choosing a winner based on the highest open rate or highest click rate amongst variations.

The updated A/B testing experience offers improved performance analysis, efficiency, and visibility, including the ability to automatically choose test winners. You can more efficiently run multiple tests for the same message at different times, and you will have the ability to run multiple tests for the same message at different times.

Perform a flow email A/B test

Create a new test

  1. To set up A/B testing for an individual email in any flow, click on the email within the flow builder.

  2. Click Create A/B Test in the upper right-hand corner of the sidebar.
    In the flow builder, the option to 'Create A/B Test' will appear in the sidebar when clicking on a message.

When you create an A/B test for an existing flow email, two new copies of the email are created. If the flow email is live and has already been sending, when the A/B test is published the analytics for each of these copies are tracked separately. We do not continue tracking analytics for the original email once you have added a variation.

Edit variations

After creating a new test, you will be taken directly to the A/B Test tab of the Manage Content window. Every test will automatically include 2 variations. When you click on a variation, you can edit the variation name, subject line, preview text, sender name, and sender email address.

  1. To edit the content of the email, click the Edit Email button above the email preview on the right.
    When viewing the configuration options for a specific variation, there is an 'Edit Email' button in the top right.

Add a new variation

If you’d like to add another variation, you must clone one of the existing 2 variations. 

  1. Click the clone icon to the right of the variation you want to base your new variation on.
    The clone button is found to the right of the variation name.
  2. This will create a new variation which you can modify to be distinct from the original 2 variations.
    The cloned variation will appear below the original variations.

Delete a variation

An A/B test must have at least 2 variations, so a variation can only be deleted when there are 3 or more. Additionally, variations can only be deleted while a test is in Draft status. Deleting variations is disabled for live A/B tests.

  1. To delete a variation, click on the trashcan icon to the right of the variation you want to delete.
    The delete button is is depicted with a trashcan icon and is found to the right of the clone button.

Choose the weight of each variation

At the bottom of the A/B Test tab, you will see the Test Settings section where you can edit the weight of each variation – this will establish the percentage of recipients that will receive each variation. 

It's generally recommended to weigh each variation equally, but you can change the percentages to any amount you’d like.

You can also select the option to automatically choose distribution. This will set the distribution to be equal for each variation at first. Over time, if one of the variations performs better than the others, Klaviyo will automatically adjust the distribution to direct more recipients to the better performing variation while the test is still running.

In the test setting section, you can configure distribution, winner selection, and notes for the A/B test.

Configure winner selection

Tests will default to automatically choose a winner based on click rate, but this can be changed.

  1. In the Test Settings section at the bottom of the A/B Test tab, you can also configure winner selection by clicking on the Configure link under Winner Selection.
    Under the Winner Selection label, there is a link to open the Winner Selection modal.

  2. A modal will appear in which you can change the Winning Metric to Open Rate instead.

    Using Click Rate is recommended since many factors such as Apple Mail Privacy Protection (MPP) can cause inflated open rates. For complete information on MPP opens, visit our iOS 15: How to Prepare for Apple’s Changes guide.
    The winner selection modal has options for selecting the winning metric (Click Rate/Open Rate) as well as turning on/off automatic winner selection.
  3. In the Automatic Winner Selection section, there are 2 options:
    1. End test if the winning metric is determined to win with high statistical certainty (statistical significance): this option is checked by default. Uncheck the box next to this option if you’d like to manually end the test instead or choose the option to end the test after a specific date.
    2. Set a date limit: this option allows you to select a specific date to end the test automatically.

  4. If you select both options, the test will end if statistical significance is reached or if the specified date is reached depending on which occurs first.

  5. Click the Save Settings button to confirm your configuration.

Publish a test

  1. Once you have your variations configured, you are ready to publish your test. From the A/B Test tab, click the Publish Test button in the upper right.
    The Publish Test button is in the upper right of the A/B Test tab.

  2. A modal will appear asking you to confirm the publication. Click the Publish Test button to confirm.
    A modal will appear asking you to confirm that you want to publish the test.

Delete a test

  1. If you wish to delete a test, from the A/B Test tab, click the arrow next to the Publish Test button and select Delete.
    Clicking the arrow next to the Publish Test button will reveal the Delete option.

View A/B test results

  1. You can view the results of a test while it is running or after completion by clicking the email within the flow builder and clicking the View A/B Test button.
    The View A/B Test button is in the left sidebar of the flow builder and will appear when a test has been created for a flow message.

This will take you to the A/B Test Results page where you can view the following metrics:

  • Recipients
    How many people received the email
  • Opens
    What percentage/number of people opened the email 
  • Clicks
    What percentage/number of people clicked through the email
  • Conversion Metric
    Total recipients who triggered the conversion metric the selected conversion metric for the email such as Placed Order

The A/B test results page has a chart for measuring performance as well as statistics related to open rate, click rate, and conversion metrics.

Manually choose a winner

  1. From the A/B Test Results page you can manually choose a winning variation for the test by clicking the Choose Winner button in the top right corner of the page.
    The Choose Winner button is in the top right of the A/B test results page.

  2. A modal will appear asking you to choose a winner from each of your existing variations. Select the variation you’d like to win from the dropdown then click the Choose Winner button.
    A modal appears with a dropdown with each message variation to choose as a winner.

Changing a message’s status from live to draft status will end the test and automatically choose a winner based on the winning metric – the first variation will be chosen if there is a tie.

Additional resources

Learn more about A/B testing in flows:

x
Was this article helpful?
10 out of 12 found this helpful