How to review your email A/B test results for campaigns

read
Last updated at:

You will learn

Learn what you can find on the A/B test results page and what each of these indicators mean.

A/B testing campaigns can provide valuable insight into your audience. Klaviyo makes it simple to determine what changes you should make to your sending strategy. Not only will you know the winning variation for an A/B test, but also other important factors, like the win confidence percentage and winning metric details. 

A/B test results

To see the results of an A/B test: 

  1. Navigate to the campaign you are testing
  2. Select the A/B Test Results tab of the main campaign report
    The A/B test results tab in a campaigns overview report

Test overview

In the Test Overview section, you’ll see a snapshot of the test results, including:

  • Winning variation
    Once a test is complete, the variation that won 
  • Win confidence
    How likely it is that the winning variation truly had the highest open, click, or placed order rate after taking random chance into account
  • Statistical significance
    Whether the winning variation won by a sufficient margin to be considered statistically significant
  • Status
    Whether the A/B test is ongoing (i.e., collecting analytics) or complete
  • Winning metric
    Which metric was used to determine the winning variation (i.e., open rate, click rate, or placed order rate)
  • Test size
    The percentage of message recipients who were part of the testing pool 

Understanding win confidence and statistical significance

Say, for instance, you want to see if adding emojis to your subject line impacts your open rate with your newsletter list. If you only send the test to 2 subscribers and the emojis have a better open rate, you can’t say conclusively that the rest of your newsletter list will feel the same way. If instead you tested with 100 people in your newsletter list and found that emojis have a substantially higher open rate, you would feel much more confident that using emojis in your subject lines is better for your audience.

Under the win confidence, you’ll see whether or not the variation shown is statistically significant. If it is deemed significant, this means that the variation has a high win probability (i.e., 90% or higher). A test might also be promising, inconclusive, or not statistically significant. Learn more about statistical significance in Klaviyo

Additional resources

x
Was this article helpful?
37 out of 52 found this helpful