How to A/B test SMS campaigns

Like with email, there are several best practices for A/B testing SMS:

• Test 1 aspect at a time.
• Include a link and use the Klaviyo link shortener.
• For MMS: keep any images or GIFs under 600 KB.
• Optimize your CTA and message text.

What to test 

Here are a few ides for what to A/B for SMS: 

• MMS versus SMS
  SMS tends to perform the same or better than MMS. Since MMS costs more credits, it's worth finding out if sending MMS messages makes sense for your brand. 
• Emojis
  Like MMS, SMS without emojis tend to have the same or better performance. In addition, emojis shorten the character limit for SMS from 160 to 70, often leading to you needing to use more credits for that send. 
• Message length
  Shorter SMS messages also tend to have higher engagement, but it's worth testing how your subscribers respond to longer-form SMS messages. 
• Language (in a certain area/country)
  Depending on your audience, you may want to test which language performs best for a certain segment, region, or country. This can be helpful if there are multiple official languages for a country (like in Switzerland) or when determining whether to send in a different language to a certain segment or area (e.g., Montreal). 

You should also test different types of SMS campaigns; e.g., product promotion, newsletter, sale, etc. What performs well for one type of message may not perform well for another. Further, a campaign might yield different results than a flow message, so do not apply the results of an A/B test to every message you send.

A/B test SMS messages

Regardless of whether you want to test message content or send times, start by following these steps: 

1. Click Campaigns > Create > SMS.
2. Choose your campaign name, recipients.
3. Click Next.
4. In the Message content box, craft your message (you will not be able to create an A/B test until you add content).
5. Click Create A/B Test to create a copy of your existing message (including any images, GIFs, personalization, etc.)
   
6. Next, decide if you want to:
   1. A/B test the SMS content 
   2. A/B test SMS send times 

Test SMS content

1. Click the dropdown menu for the variations.
2. Choose the variation you want to change for the test.
3. Change only one aspect from the original (e.g., if you add an image, do not change anything else about the message).
4. Click Continue to Test Settings in the upper right.
5. Review your A/B test settings (click the link to skip to that section).

Test SMS send times

1. Segment your audience by location or country.
2. Choose which segment you want to A/B test.
   Note: you should only send the A/B test to one of these segments so that you don’t risk messaging someone too late at night or early in the morning. Klaviyo warns you if your campaign will potentially violate quiet hours.
3. Keep the message content exactly the same.
4. Select Continue to Test Settings in the upper right.
5. In the A/B test settings page, click Test send times.
   
   
6. Choose the times you want to A/B test.
7. Configure the rest of the A/B test’s settings, which is discussed in the next section.

Select a test strategy

If your account qualifies, decide on your testing strategy from the following options:

• Winning variation (Standard A/B test)
• Personalized variations for each recipient

The difference between winning variation and personalized variations

Both test strategy options will send variations to a test group composed of a percentage of the campaign’s total recipients and test message success based on a metric of your choosing, e.g., open rate. However, there are some key differences for how each strategy sends messages to the rest of the recipients after the test period is over:

• Winning variation - the default option for A/B tests, which will determine 1 winning variation and send it to the rest of the campaign's recipients after the test period is over.
• Personalized variations - uses AI to search for patterns amongst the test recipients who interact with each variation. After the testing period is over, Klaviyo will predict which variation will perform better for each recipient and send the rest of the recipients their preferred variation.

Personalize variations for each recipient 

If you’d like to personalize which variation each recipient receives on an individual basis, select Personalized variations for each recipient. 

Klaviyo will use information about each profile to determine which variation is most likely to succeed in converting that profile. This profile information includes, but is not limited to:

• Historical engagement rates
• Customer lifetime value (CLV)
• Location

For example, if you have chosen open rate as your winning metric and profiles in the test group with a CLV of 100 or more open variation A more frequently, but profiles with a CLV of less than 100 open variation B more frequently, the rest of the campaign's recipients will receive SMS based on which variation they are most likely to open according to their CLV. This is a simplified example, as Klaviyo will use many data points to determine personalized variations.

Adjust an A/B test’s settings (optional)

You can view the settings for your test and adjust:

• The test's winning metric (click rate or placed order rate)
• What percentage of people receive each variation
• The test duration

Once a winning variation is selected, the winner will be sent to the rest of the audience.

Additional resources

• Best practices for A/B testing
• Understand statistical significance in Klaviyo
• How to A/B test via flow branches
• How to review your A/B test results for campaigns
• Mastering A/B Testing: Klaviyo’s Proven Tips & Tricks to Boost Revenue