Understanding statistical significance in flows

Last updated at:

You will learn

Learn when an A/B test’s results are considered to be statistically significant in Klaviyo flows. This article will explain statistical significance for flows and when it is decided.

Statistical significance is when Klaviyo is mathematically able to determine whether a variation will produce improved performance. You can configure a flow A/B test to automatically choose a winning variation once the test results are considered statistically significant.

A/B testing is key to driving better engagement and improving your customer relationships through a data-driven approach. It’s important to understand whether the test can be considered statically significant or not. For example, if a flow email has 2 variations, and variation A has a click rate of 15% while variation B has a click rate of 14%, how do you know for sure if variation A or B performed better?

How Klaviyo determines statistical significance in flows

For flows, Klaviyo observes both the number of people who received a message and the win probability, which is how likely a variation will yield better results based on how well it outperforms the other variation(s).

A flow message variation is considered statistically significant when:

  • At least 500 recipients have received each variation
  • A variation has at least 90% win probability

Win probability is calculated based on the metric you chose when configuring your A/B test. By default, this metric is the click rate of the message you’re testing. In the Automatic Winner Selection section, you have the option to automatically end the test once a message variation is determined to win based on the configured metric, as well as the option to end the test once a specific date is reached. You can select either or both of these options. If both are selected, the test will end based on which is reached first, statistical significance or the specified date.

For more details, see our article on how to A/B test a flow email.

The winner selection modal has options for selecting the winning metric (Click Rate/Open Rate) as well as turning on/off automatic winner selection.

When viewing the results for a currently running test, you will see the win probability at the top of the left sidebar and whether or not the test results are statistically significant in the Winning Metric section. For more details, see our article on how to review email A/B test results for flows.

The overview section for a currently running test

Additional resources

Check out our article on best practices for A/B testing

Learn about other A/B tests you can run:

Was this article helpful?
2 out of 3 found this helpful