How to A/B test flow SMS messages

Learn how to A/B test your SMS messages in flows and find inspiration on what to test.

How to test your SMS messages

The best approach for testing your SMS flow message depends on what you want to test.

Click on what you want to test for details on how to do it.

• Test message content
  • MMS versus SMS
  • Emojis
  • Discounts
  • CTAs
  • Personalization
• Timing or number of messages
• Message channel
  • Email versus SMS
  • SMS versus push

Test message content

1. Navigate to the flow where you want to A/B test an SMS.
2. If you haven’t already, drag an SMS into the flow.
3. In the left sidebar, click either Configure Content or Edit.
4. Write new content or edit existing copy (you will need to have something typed in the message box before A/B testing is available).
5. Click Create A/B test.
   

6. In the box for message B, click Edit Message.
   

7. Change the content in the message editor box.
   As a best practice, only test 1 element of a message at a time. For instance, you should either make the CTA different, add an image, or include an emoji; don’t do all 3 in a single message.
8. Optional: To add more variations, click the Clone button next to one of the existing variations. Then, click into the new variation and edit it.
   Note that you can have up to 25 variations.
9. Click Save and Exit.
10. Optional: adjust the settings for the A/B test, such as:
    • Changing the sending distribution (e.g., the percent of people who will receive variation A versus variation B)
    • Configuring the winner selection settings
    • Adding notes for the A/B test (e.g., explain what you’re testing)
      

1. Click Publish test, then confirm by clicking Publish test in the confirmation window. If the SMS message is still in Draft status, return to the flow and change its status to Live when you’re ready to send it; publishing the test alone does not make a draft message live.

Test the timing or number of messages

To test either the timing or number of messages in a flow, follow these steps:

1. Navigate into the flow.
2. Drag a conditional split and place it where you want to test.
3. In the left sidebar, click the Select the condition dropdown menu.
4. Choose Random sample.
5. Select your percentage.
   Note that the percentage represents how many should go down the YES path. If you choose 10%, only 10% will go down the YES path, while 90% of people will go down the NO path.
   

6. Drag in your messages and time delays onto each path.
7. Rejoin the split.

When testing the timing of messages, make sure the content is exactly the same on each path.

If you’re testing the number of messages, keep the timing and content of the messages as similar as possible. This is to limit the number of variables that can affect the results.

Test message channel

Test the message channel (e.g., email or SMS) by following these steps:

1. Navigate into the flow.
2. Drag a conditional split and place it where you want to test.
3. In the left sidebar, click the Select the condition dropdown menu.
4. To compare an SMS against an email, choose can receive SMS marketing.
5. Leave the condition as Person can receive SMS marketing.
6. Click Save.
7. Add an email to the NO path. This path includes people who cannot receive SMS marketing; it does not necessarily consist of people who can receive email marketing.
8. Place a conditional split on the YES path.
9. In the left sidebar, click the Select the condition dropdown menu.
10. Choose Random sample.
11. Select your percentage.
    Note that the percentage represents how many should go down the YES path. If you choose 10%, only 10% will go down the YES path, while 90% of people will go down the NO path.
    

12. Place an SMS on the YES path and configure the message.
13. Copy the email on the first NO path and drag it to the NO path below the random sample.
14. Rejoin the split paths.

Determine a winner when using splits

When you test with conditional splits, review the results of the messages on each path yourself. A conditional split does not choose a winner or show statistical significance for the paths as a flow message A/B test does. Use flow analytics to review each message’s performance over the same time period.

When testing the number of messages, review the results of all messages on each path rather than only the first message. Do not treat a sum of message-level counts as the number of unique people who responded: the same person can receive more than one message on a path.

Additional resources