Overview
When A/B testing individual emails, you can test things like subject line, discounts, and email content. You can even figure out whether, say, plain-text emails perform better than image-heavy HTML emails. Keep in mind that A/B testing cannot currently be used for SMS flow messages.
Note that you can also A/B test flow branches. For details on that, check out How to A/B Test Flow Branches.
Flow email A/B testing is the process of:
- Creating a single flow email and scheduling this email to send after a certain number of hours or days, just as you normally would.
- Configuring two or more email variations for this single flow email — adjusting the subject line or core email content for each variation.
- Selecting the weight of each variation to establish the percentage of recipients that will receive each email.
- Observing the performance of each variation over time to determine which should be considered the "winner".
In this article, we show you how to perform an A/B test for individual flow emails
Perform a Flow Email A/B Test
To set up A/B testing for an email in any flow, select the email card and click Add Variation in the upper right-hand corner of the sidebar. You will be taken directly to a Manage Variations window. Here, you can edit the subject line and/or content of each variation by clicking Edit Variation.
To add another variation, click Add Variation in the top menu or alternatively click the Actions dropdown for an existing variation and select Duplicate Variation.
Lastly, choose the weight of each variation -- this will establish the percentage of recipients that will get each one. It's common practice to weight each variation equally, which you can accomplish quickly by clicking Equalize Weights.
If you've just configured a new email and are in the Message Content view, you can quickly add a variation without returning to the main flow canvas by clicking the Add Variation link below your configured message. Clicking this link will bring you directly to the Manage Variations window where you can edit the subject line and/or content of each variation.
After setting your flow email with A/B testing live, you can monitor the performance of each variation across the following metrics:
- Recipients: How many people received the email
- Opens: What percentage/number of people opened the email
- Clicks: What percentage/number of people clicked through the email
- Placed Order:
- Total of recipients who placed an order
- Total dollar amount that was made from each email
9/20/21 Update: with the upcoming release of iOS 15, macOS Monterey, iPadOS 15, and WatchOS 8, Apple will change the way that we receive open rate data on your emails by prefetching our tracking pixel. With this change, it’s important to understand that open rates will be inflated.
Thus, if you are triggering any flows off of opens themselves, we suggest moving to expand that trigger definition to include other engagement criteria like clicks or product views. It may also be useful to look at the number of iOS-specific users within your data to understand how your open rates will change.
For complete information on ways to potentially mitigate these Apple open changes, visit our iOS 15: How to Prepare for Apple’s Changes guide.
Once you determine that one variation is outperforming the others over a time period that you deem significant, you can delete the other variations so that 100% of your recipients will receive the tested winner. Alternatively, you can swap out the underperforming variation with a new test and repeat the cycle.
Additional Resources