How to A/B Test a Flow Email

read
Last updated at:

Overview

When A/B testing individual emails, you can test things like subject line, discounts, and email content. You can even figure out whether, say, plain-text emails perform better than image-heavy HTML emails. Keep in mind that A/B testing cannot currently be used for SMS messages. 

Note that you can also A/B test flow branches. For details on that, check out How to A/B Test Flow Branches. 

Flow email A/B testing is the process of:

  1. Creating a single flow email and scheduling this email to send after a certain number of hours or days, just as you normally would.
  2. Configuring two or more email variations for this single flow email — adjusting the subject line or core email content for each variation.
  3. Selecting the weight of each variation to establish the percentage of recipients that will receive each email.
  4. Observing the performance of each variation over time to determine which should be considered the "winner".

In this article, we show you how to perform an A/B test for individual flow emails

Perform a Flow Email A/B Test

To set up A/B testing for an email in any flow, select the email card and click Add Variation in the upper right-hand corner of the sidebar. You will be taken directly to a Manage Variations window. Here, you can edit the subject line and/or content of each variation by clicking Edit Variation.

To add another variation, click Add Variation in the top menu or alternatively click the Actions dropdown for an existing variation and select Duplicate Variation.

When you add a variation to a flow email that has already been sending, two new copies of this email are created, and the analytics for each of these copies is tracked separately. We do not continue tracking analytics for the original email once you have added a variation. The Overview report will include historical information from the original email, along with any new variations. You can view analytics for a specific variation by clicking into the Variations tab, or by checking the box to show all variations when exporting.

Lastly, choose the weight of each variation -- this will establish the percentage of recipients that will get each one. It's common practice to weight each variation equally, which you can accomplish quickly by clicking Equalize Weights.

2018-01-12_22-51-35.gif

If you've just configured a new email and are in the Message Content view, you can quickly add a variation without returning to the main flow canvas by clicking the Add Variation link below your configured message. Clicking this link will bring you directly to the Manage Variations window where you can edit the subject line and/or content of each variation.

AddVariation.png 

After setting your flow email with A/B testing live, you can monitor the performance of each variation across the following metrics:

  • Recipients: How many people received the email
  • ​Opens: What percentage/number of people opened the email 
  • Clicks: What percentage/number of people clicked through the email
  • Placed Order: 
    • Total of recipients who placed an order
    • Total dollar amount that was made from each email

Once you determine that one variation is outperforming the others over a time period that you deem significant, you can delete the other variations so that 100% of your recipients will receive the tested winner. Alternatively, you can swap out the underperforming variation with a new test and repeat the cycle.

Additional Resources

 

x
Was this article helpful?
113 out of 139 found this helpful