A/B Testing a Flow

read

Overview

There are two key ways you can A/B test flows in Klaviyo:

  1. A/B test individual emails within a flow
    When A/B testing individual emails, you can test things like subject line and email content. This is similar to the process of A/B testing a campaign, since the test only applies to a single email.
  2. A/B test different branches within a flow
    You can A/B test different flow branches using the Split component. This allows you to test things like timing and number of emails within a particular flow.

A/B Test Individual Flow Emails

A/B testing individual emails allows you to easily test different subject and email content to help you refine your messaging. Flow email A/B testing is the process of:

  1. Creating a single flow email and scheduling this email to send after a certain number of hours or days, just as you normally would
  2. Configuring 2 or more email variations for this single flow email -- adjusting the subject line or core email content for each variation
  3. Selecting the weight of each variation to establish the percentage of recipients that will receive each email
  4. Observing the performance of each variation over time to determine which should be considered the "winner"

Once you determine that one variation is out-performing the others over a time period that you deem significant, you can delete the other variations so that 100% of your recipients will receive the tested winner. Alternatively, you can swap out the under-performing variation with a new test and repeat the cycle.

Set Up a Flow Email A/B Test

To set up A/B testing for an email in any flow, select the email card and click Add Variation in the upper righthand corner of the sidebar. You will be taken directly to a Manage Variations window. Here, you can edit the subject line and/or content of each variation by clicking Edit Variation.

To add another variation, click Add Variation in the top menu or alternatively click the Actions dropdown for an existing variation and select Duplicate Variation.

Lastly, choose the weight of each variation -- this will establish the percentage of recipients that will get each one. It's common practice to weight each variation equally, which you can accomplish quickly by clicking Equalize Weights.

2018-01-12_22-51-35.gif

If you've just configured a new email and are in the Message Content view, you can quickly add a variation without returning to the main flow canvas by clicking the Add Variation link below your configured message. Clicking this link will bring you directly to the Manage Variations window where you can edit the subject line and/or content of each variation.

AddVariation.png 

After setting your flow email with A/B testing live, you can monitor the performance of each variation across the following metrics:

  • Recipients: How many people received the email
  • Opens: What percentage/number of people opened the email 
  • Clicks: What percentage/number of people clicked through the email
  • Placed Order: 
    • Total of recipients who placed an order
    • Total dollar amount that was made from each email

Flow Email A/B Testing Examples and Best Practices

A few examples of how Klaviyo customers use A/B testing:

  • Experimenting with new subject lines or types of emails in a very low-risk way. Because A/B tests allow you to send unique variations to a limited subset of people, this is a great way to try something new and get real data to measure how well the idea worked (rather than having to debate the anticipated outcome without metrics to back up your assumptions).
  • Understanding the right discount to offer and whether they are necessary. Because you can easily test completely different email content, A/B tests are a great way to figure out if you really need to offer 20% off or if you can instead get away with 10% or no offer at all.
  • Figuring out whether plain text emails perform better than image-heavy HTML emails. Many of our customers test out both very short personal emails as well as traditional emails with images and complex layouts. Both work well for different purposes, but A/B testing will give you actual numbers to analyze when considering which approach is best.

A/B Test Flow Branches

A/B testing different branches within a flow can be a great way to test the timing or number of emails within a flow. A/B testing flow branches is a process of:

  1. Building out the control branch that you would like to A/B test as a starting point
  2. Dragging in a Conditional Split component where you would like the branch testing to begin
  3. Configuring the Split to divide the different tracks using the "Random sample" condition
  4. Building out the content of the test branch

Set Up a Test Branch

First, navigate to the flow you would like to A/B test in the Flows tab. Then, drag a Conditional Split component where you would like to start your A/B test. For example, you may want to test sending an additional email to one branch of your welcome series. To do this, you would drag a Split to the end of your flow, add a time delay, and then configure the email content.

2018-05-31_14-52-49.png

When configuring the Split, select Random sample as the condition. Here, you will be prompted to select what percentage of your audience will go down the YES path, which you can think of as the control. For an even A/B test, select 50%. 

Once you have your Split configured, you can build out the content of your test branch. Depending on what you're testing, this could be several emails or different timing from what you have in your control branch.

Flow Branch A/B Testing Examples and Best Practices

When running an A/B test, bear in mind that testing more than one variable at a time can skew results and make it difficult to determine how to attribute any differences in revenue, open rates, etc. For this reason, it's a best practice to test variables one at a time after you've determined a winner.

Some examples of when you might want to A/B test flow branching include:

  • Testing the number of emails in your welcome series. You may want to test adding an additional email at the end of your welcome series to see if you can convince more customers to buy.
  • Testing discounts. Test a discount vs. no discount to see which email converts better, or test varying degrees or types of discounts.
  • Testing email timing for your abandoned cart flow. You may be interested in testing whether an email sent sooner converts better -- you can do this by adding a Split component to the beginning of the flow.
Was this article helpful?
5 out of 6 found this helpful