You can A/B test different flow branches using the conditional split component. This allows you to test things like timing, discounts, and number of emails within a particular flow. In this article, we run through how to set up and end a A/B test for a flow branch.
You can also A/B test a single flow email. Read How to A/B Test Flow Emails for guidance.
Set Up a Test Branch
First, navigate to the flow you would like to A/B test in the Flows tab. Then, drag a conditional split to where you would like to start your A/B test. For example, you may want to test sending an additional email to one branch of your welcome series. To do this, you would drag a split to the end of your flow, add a time delay, and then configure the email content.
When configuring the split, select Random sample as the condition. Here, you will be prompted to select what percentage of your audience will go down the YES path, which you can think of as the control. For an even A/B test, select 50%.
Once you have your split configured, you can build out the content of your test branch. Depending on what you're testing, this could be several emails or different timing from what you have in your control branch.
When running an A/B test, bear in mind that testing more than one variable at a time can skew results and make it difficult to determine how to attribute any differences in revenue, open rates, etc. For this reason, it's a best practice to test variables one at a time after you've determined a winner.
9/20/21 Update: with the upcoming release of iOS 15, macOS Monterey, iPadOS 15, and WatchOS 8, Apple will change the way that we receive open rate data on your emails by prefetching our tracking pixel. With this change, it’s important to understand that open rates will be inflated.
Thus, if you are triggering any flows off of opens themselves, we suggest moving to expand that trigger definition to include other engagement criteria like clicks or product views. It may also be useful to look at the number of iOS-specific users within your data to understand how your open rates will change.
For complete information on ways to potentially mitigate these Apple open changes, visit our iOS 15: How to Prepare for Apple’s Changes guide.
End the A/B Test
After deciding which branch is better for your audience, click on the conditional split. To have all of the recipients go down the Yes path, set the percentage to 100% (as shown in the example below); If you want everyone to go down the No path, set it to 0%. This way, those who were already in the Waiting queue still receive the old message rather than being taken out of the flow like they would if you delete the branch. Once they do, set the branch's emails into draft mode.