Understanding statistical significance in Klaviyo campaigns

You will learn

Learn how to tell whether an A/B test is statistically significant. For instance, if campaign variation A has an open rate of 15% and variation B has an open rate of 14%, how do you know which variation performed better?

Note about Apple Mail Privacy Protection

With the release of iOS 15, macOS Monterey, iPadOS 15, and watchOS 8, Apple Mail Privacy Protection (MPP) changed the way that we receive open rate data on your emails by prefetching our tracking pixel. With this change, it’s important to understand that open rates will be inflated.

MPP opens are included when an A/B test uses open rate as its winning metric. Statistical significance does not establish that recipients intentionally opened the email. Consider click rate if MPP makes open-rate results difficult to interpret. To understand how MPP affects your audience, create a custom report that includes an MPP property. You can also identify these opens in your individual subscriber segments.

For complete information on MPP opens, visit our iOS 15: How to Prepare for Apple’s Changes guide.

Categories of statistical significance

Statistical significance indicates strong evidence that one variation performs better than the others for your chosen winning metric. For campaigns, Klaviyo evaluates both audience size and win probability: the estimated likelihood that a variation has the highest rate for that metric, based on the test data.

When A/B testing, you should avoid any factor that can significantly influence your audience; for instance, don’t retest near a holiday weekend, when your audience is much more likely to be looking for your emails.

When it comes to A/B testing campaigns, there are 4 categories for statistical significance:

• Statistically significant
• Promising
• Not statistically significant
• Inconclusive

To review your results, open the campaign report and select A/B test results. After the test is complete, look under Win probability. If both Test period results and Continuous results are available, select Test period results to assess the data collected before the winner was selected. Continuous results includes later activity from the test recipients.

Statistically significant

The Statistically significant label on your A/B test means that one variation is highly likely to outperform the other option(s) for the chosen metric, based on the test data. You can use what you learned to guide future sends, but the result does not guarantee the same outcome with a different audience, message, or send time.

For Klaviyo campaigns, an A/B test result is deemed statistically significant when:

• Each variation has more than 50 successful deliveries for email, or more than 50 recipients for SMS.
• The win probability is greater than 90%.

These criteria combine audience size with confidence in the result for your chosen winning metric: open rate, click rate, or placed order rate. The displayed win probability is rounded, so use the result label or alert when interpreting a value near a cutoff.

For example, say you’re testing whether an emoji in the subject line affects open rate. The results show that the variation with the emoji won and has the Statistically significant label. This supports using the emoji in similar sends to that audience, but it does not establish that emojis will consistently improve open rates in every campaign.

However, you should always use your best judgment when acting on the results of the test. If you’re sending out a more serious or somber message, for instance, an emoji might not be appropriate to use.

Promising

When the results are promising, one variation looks like it’s performing better than the other(s), but the evidence isn’t strong enough from the test by itself. Promising results do not have a significance label. For a completed test, an alert recommends repeating the test. If Test period results is available, look there for this alert. If a different variation leads in Continuous results, an alert about the change in winner appears instead.

For a promising result, perform another similar A/B test to check whether the result holds. Going back to the emoji example, repeated promising results can suggest a pattern, but they do not by themselves confirm statistical significance. Use them to guide further testing rather than assume emojis will improve every campaign.

For campaigns, an A/B test result is considered promising when:

• Each variation has more than 50 successful deliveries for email, or more than 50 recipients for SMS.
• The win probability is greater than 75% and no more than 90%.

Not statistically significant

A not statistically significant result means the test does not provide strong enough evidence that the leading variation performs better. A variation can still be selected as the winner even when the result is not statistically significant.

For our example, this may mean that the variation with the emoji beat the no-emoji variation(s), but only by a very slight amount; not enough for the test result to be meaningful.

In this case, we recommend retesting this factor 2 or 3 more times and keeping a close eye on the results.

If you continue to see that the test result is deemed not statistically significant, do not continue to test; instead, move on to A/B testing a different topic. If a test falls into this category, a Not statistically significant label will appear on the A/B test results page.

For campaigns, each variation must have more than 50 successful deliveries for email, or more than 50 recipients for SMS, before a result can be classified as not statistically significant. Once that minimum is met, either of these conditions qualifies:

• The win probability is 75% or lower, and every variation has more than 1,800 successful deliveries for email, or more than 1,800 recipients for SMS.
• The win probability is below 60%, and the difference in the winning metric between the leading and second-place variations is less than 4 percentage points.

Inconclusive

If a test is inconclusive, there is not enough information to determine whether a variation performs better. If any variation has 50 or fewer successful deliveries for email, or 50 or fewer recipients for SMS, the result is inconclusive regardless of win probability. Results that do not meet any of the other categories’ criteria are also inconclusive. You can visualize the categories using the decision tree in the next section. Inconclusive tests do not show a statistical significance label on the results page.

In this case, you may want to expand your audience for any follow-up tests. If you have a small group that you’re testing with, interpret the results as you see fit and retest to check that what you’ve found is accurate. Further, focus on learning about your subscribers through other means; e.g., surveys, polls, etc.

Statistical significance decision tree

The following decision tree shows when an A/B test will fall into each category. The green lines indicate that the answer is “yes” while red represents “no.”

Additional resources