Understanding statistical significance in flows
Learn when an A/B test’s results are considered to be statistically significant in Klaviyo flows. This article will explain statistical significance for flows and when it is decided.
Statistical significance means the test provides enough evidence that one variation is likely to perform better than the others on your chosen winning metric. It does not guarantee that the variation will perform better in the future. You can configure a flow A/B test to automatically choose a winner when the results become statistically significant.
A/B testing is key to driving better engagement and improving your customer relationships through a data-driven approach. It’s important to understand whether the test can be considered statistically significant or not. For example, if a flow email has 2 variations, and variation A has a click rate of 15% while variation B has a click rate of 14%, how do you know for sure if variation A or B performed better?
How Klaviyo determines statistical significance in flows
For flows, Klaviyo observes both the number of people who received a message and the win probability, which is how likely a variation will yield better results based on how well it outperforms the other variation(s).
A flow message variation is considered statistically significant when:
At least 10 recipients across all variations have completed the action measured by the winning metric, such as clicking when the winning metric is click rate.
- More than 500 messages have been delivered for each variation.
- A variation has greater than 90% win probability.
- The leading variation’s likely performance range has less than 10% overlap with the range for each other variation.
Win probability is calculated based on the metric you chose when configuring your A/B test. By default, this metric is the click rate of the message you’re testing. Under Winner Selection in Test Settings, click Configure to choose the winning metric and automatic ending criteria. In Automatic winner selection, you can select End test by statistical significance, Set a date limit, or both. If you select both, the test ends when either condition is reached first. If the date limit is reached before statistical significance, Klaviyo selects the variation with the highest observed rate for the winning metric; reaching the date limit does not mean the results are statistically significant.
For more details, see our article on how to A/B test a flow email.
To view results for a running or completed test, click the flow email in the flow builder, then click View Test. In Test Overview, you can see the win probability and whether the results are statistically significant. For a running test, the leading variation is the one currently most likely to win; a completed test shows its selected winner.
For more details, see our article on how to review email A/B test results for flows.
Additional resources
- Best practices for A/B testing reference
Learn best practices for using Klaviyo's A/B testing features to help optimize your communication with customers.
Learn about other A/B tests you can run:
- How to A/B test a flow email
Learn how to use Klaviyo’s A/B testing experience to test individual emails in a flow, including things like subject line, discounts, and email content, etc. You can even use A/B tests to determine if plain-text emails perform better than image-heavy HTML emails.
- How to A/B test an email campaign
Learn how to set up and run an A/B test for a campaign, how to read the test results, and a few use cases for A/B testing campaigns. Klaviyo's A/B testing feature for campaigns allows you to easily test different subject lines, message content, and send times so you can better understand what works best for your audience.
- How to A/B test a sign-up form
Learn how to use A/B testing to evaluate the performance of various elements of your sign-up forms. By effectively analyzing the results, you can ensure that your forms resonate with site visitors and contribute to list growth.