How to A/B test SMS campaigns

You will learn

Learn how to A/B test SMS campaigns and what aspects to test in this article.

With A/B testing for Klaviyo SMS, you can gain insight into your audience and use that information to optimize your messages and send times. This article covers campaign tests. You can also test your SMS flow messages using splits.

Best practices for A/B testing SMS

Like with email, there are several best practices for A/B testing SMS:

• Test 1 aspect at a time.
• Include a link and use the Klaviyo link shortener.
• For MMS: keep any images or GIFs under 600 KB.
• Optimize your CTA and message text.

What to test

Here are a few ideas for what to A/B test for SMS:

• MMS versus SMS
  SMS tends to perform the same or better than MMS. Since MMS costs more, it's worth finding out if sending MMS messages makes sense for your brand.
• Emojis
  Like MMS, SMS messages without emojis tend to have the same or better performance. In addition, emojis shorten the character limit for SMS from 160 to 70, often leading to you needing to use more message segments (higher cost) for that send.
• Message length
  Shorter SMS messages also tend to have higher engagement, but it's worth testing how your subscribers respond to longer-form SMS messages.
• Language (in a certain area/country)
  Depending on your audience, you may want to test which language performs best for a certain segment, region, or country. This can be helpful if there are multiple official languages for a country (like in Switzerland) or when determining whether to send in a different language to a certain segment or area (e.g., Montreal).

You should also test different types of SMS campaigns; e.g., product promotion, newsletter, sale, etc. What performs well for one type of message may not perform well for another. Further, a campaign might yield different results than a flow message, so do not apply the results of an A/B test to every message you send.

Before you begin

Klaviyo gives you the option to either test message content (e.g., the call-to-action, emojis, and images/GIFs) or send times.

You can't create an A/B test while content personalization is enabled for the message.

For a send-time test, first segment your audience by location or country. Send-time tests use your account's time zone, so choose times that are appropriate for recipients in the segment.

Select that segment when you choose the campaign's recipients.

A/B test SMS messages

Regardless of whether you want to test message content or send times, start by following these steps:

1. Navigate to Campaigns and click Create campaign.
2. Enter a campaign name, choose Single channel, select a date, and choose Text message.
3. Click Continue.
4. Choose your recipients, then click Next.
5. In the Message content box, craft your message (you will not be able to create an A/B test until you add content).
6. Click Create A/B test to create a copy of your existing message (including any images, GIFs, personalization, etc.)
   

7. Next, decide if you want to:
   1. A/B test the SMS content
   2. A/B test SMS send times

Test SMS content

1. Click the dropdown menu for the variations.
2. Choose the variation you want to change for the test.
3. Change only one aspect from the original (e.g., if you add an image, do not change anything else about the message).
4. Click Next in the upper right to save the message and open the A/B test settings.
5. Select a test strategy and review your A/B test settings.

If you are A/B testing content and would like to send your message based on each recipient’s timezone, set your test size to 100%. When scheduling your campaign, you’ll have the option to choose Recipient's Local Time Zone as the timezone for your send.

Test SMS send times

1. Keep the message content exactly the same.
2. Click Next in the upper right to save the message and open the A/B test settings.
3. Under Test type, select Test send time.

4. Choose the dates and times you want to A/B test. These times use your account's time zone.
5. Review the remaining settings below. Send-time tests use 100% of the audience, so you can't adjust the test size or test duration, and no winning message is sent to a remaining audience.

Select a test strategy

If your account qualifies, decide on your testing strategy from the following options:

• Winning variation (Standard A/B test)
• Personalized variations for each recipient

The difference between winning variation and personalized variations

For content tests with a test size below 100%, both strategies send variations to a test group. SMS tests use click rate, or placed order rate for standard A/B tests when available, to measure success. After the test period, each strategy sends to the remaining recipients differently:

• Winning variation - the default option for A/B tests, which will determine 1 winning variation and send it to the rest of the campaign's recipients after the test period is over.
• Personalized variations - uses AI to search for patterns amongst the test recipients who interact with each variation. After the testing period is over, Klaviyo will predict which variation will perform better for each recipient and send the rest of the recipients their preferred variation.

Personalize variations for each recipient

If you’d like to personalize which variation each recipient receives on an individual basis, select Personalized variations for each recipient.

Klaviyo uses information about each profile to predict which variation they are most likely to click. This profile information includes, but is not limited to:

• Historical engagement rates
• Customer lifetime value (CLV)
• Location

For example, if profiles in the test group with a CLV of 100 or more click variation A more frequently, but profiles with a CLV of less than 100 click variation B more frequently, Klaviyo can use that pattern to choose variations for the remaining recipients. This is a simplified example, as Klaviyo uses many data points to determine personalized variations.

Adjust an A/B test’s settings (optional)

Review the settings for your test:

• The test's winning metric: Click rate or, for standard A/B tests, Placed order rate when your account has a mapped Placed Order metric. Personalized SMS variations use click rate.
• For content tests, the percentage of recipients in the test group. Send-time tests use 100%.
• For content tests below 100%, the test duration. If you choose placed order rate, we recommend at least 24 hours.

For content tests below 100%, Klaviyo sends the winning variation or each recipient's personalized variation after the test period, depending on your strategy. With a 100% test, everyone receives a test variation and there is no later winner send.

Review and schedule your campaign

1. Click Next to save your test settings and review your campaign. Resolve any issues shown in the review.
2. Click Schedule or send.

• For a content test, choose when to send and check the time zone. Click Schedule message to confirm a scheduled send, or Send now to send immediately.
• For a send-time test, review the dates and times for each variation, then click Schedule message.

Additional resources