Understanding the experiments section
You will learn
Learn how to analyze the success of your A/B tests using the experiments section. The experiments section brings together campaign and flow A/B tests, and offers insights into the tested elements, the marketing channels used, success rates, and other data.
Navigating to the experiments section
Owners, admins, managers, and analysts have access to the experiments section.
Navigate to Analytics > Experiments. If the page shows tabs, select A/B tests.
The Campaigns view opens by default. Select Flows to review flow A/B tests, or Campaigns to return to campaign A/B tests.
Using the search and settings options
At the top of the Campaigns or the Flows view, there are options for customizing your A/B tests view and the ability to search for a specific test.
Searching for a test
In the search field, enter a campaign name or flow test name, or a keyword in that name (e.g., “Testing BFCM welcome” or “welcome”). Results update as you type.
The search functionality is for the Campaigns and Flows views independently. In other words, your searches will be limited to results in each view.
Adjusting settings options
Use the filters at the top of each view to narrow the A/B test results. If a filter is not visible, click the filter icon to open More filters.
Settings changes only apply to the Campaigns and Flows views independently. In other words, any choices from these dropdowns will be limited to results in each view.
Timeframe settings
In the dropdown under Timeframe, you can choose to review your A/B test results from:
- Today
This includes tests with a start date today, using UTC. - Last 7 days
This includes all tests that had a start date within the last 7 days. - Last 30 days
This includes all tests that had a start date within the last 30 days. - Last 90 days
This includes tests with a start date within the last 90 days. - This year
This includes all tests that had a start date within the present calendar year to date. - Last year
This includes all tests that had a start date in the prior calendar year. For example, if it’s currently 2023, this would be all tests with start dates from 2022. - All time
This includes all tests that were run or are currently running.
The date range can include running and completed tests, as well as cancelled campaign tests.
Marketing channel settings
In the dropdown under Channel, you can choose to narrow your results from:
- Email
This option shows just email A/B tests. - Text message
This option shows just SMS A/B tests.
If your Flows view includes Push notification or WhatsApp in the Channel filter, select one to view tests for that channel.
To see tests across channels, clear the Channel filter.
Metric settings
In the dropdown under Winning metric, you can choose to narrow your results to those only containing certain winning metrics, including:
- Open rate
Only see those A/B tests using Open rate as the winning metric. - Click rate
Only see those A/B tests using Click rate as the winning metric. - Placed order rate
In the Campaigns view, only see those A/B tests using Placed order rate as the winning metric.
To see tests regardless of their winning metric, clear the Winning metric filter.
Learn more about configuring A/B tests for emails, SMS, or flows, including what winning metric to use for your particular testing needs.
Other filter settings
Use these filters to narrow the results further:
- Test type (only in the Campaigns view)
Choose Content test for A/B tests of message content or elements of your emails (e.g., different preview text and subject lines), or Send time test for tests of message send times. Clear this filter to see both types. - Status
Choose In progress to see A/B tests currently running or Completed to see tests that are finished. - Statistical significance
Choose Is statistically significant or Not statistically significant to filter by whether the test has reached statistical significance.
Click Clear to reset the search and filters in the current view. This also restores the initial Timeframe selection: Last 90 days or All time if there were no tests in that period.
Reviewing your collected A/B test results
Both views initially use Last 90 days. If no tests are found in that period, the view switches to All time. Select All time to look for older tests. The results include running and completed tests, as well as cancelled campaign tests, that match your filters. Draft tests and Smart Send Time tests are not included.
The tables list A/B tests rather than individual variations. A flow can have multiple tests. Click a campaign or test name to see its variations in more detail.
These tables contain the following columns:
- Campaign name or Test name
The Campaigns view shows the campaign name. If a campaign message has a different name, it appears after the campaign name. The Flows view shows the test name. Both views show the winning metric below the name.
Click a name to open its A/B test results and review the variations, data, and win result. - Channel
The marketing channel being tested, such as Text message (shown with a speech bubble icon) or Email (shown with an envelope icon). - Test type (only in the Campaigns view)
This will show whether the test is a content test (as indicated by the pencil icon) or a send time test (as indicated by the clock icon). - Start date
When the A/B test was originally started. - End date (only in the Flows view)
When the test ended. A dash appears if it has not ended. - Test size
How many profiles were included in the A/B test across all variations. - Win probability
How likely a variation is to outperform the other variations on the chosen winning metric.
Win probability alone does not establish statistical significance. For campaigns, each variation must have more than 50 recipients, and a variation must have greater than 90% win probability. For flow tests, Klaviyo also checks engagement and how clearly the variations’ results differ, in addition to the sample size. See statistical significance in flows and statistical significance in campaigns for the full criteria. - Status
Completed for a finished test, In progress for a running test, or Cancelled for a cancelled campaign test.
By default, the table sorts by the most recently started test. Use the arrows next to Campaign name or Test name, Channel, Start date, Test size, Win probability, or Status to sort in ascending or descending order. You can also sort by Test type in the Campaigns view and End date in the Flows view.
Additional resources
- How to A/B test an email campaign
Learn how to set up and run an A/B test for a campaign, how to read the test results, and a few use cases for A/B testing campaigns. Klaviyo's A/B testing feature for campaigns allows you to easily test different subject lines, message content, and send times so you can better understand what works best for your audience.
- How to A/B test a flow email
Learn how to use Klaviyo’s A/B testing experience to test individual emails in a flow, including things like subject line, discounts, and email content, etc. You can even use A/B tests to determine if plain-text emails perform better than image-heavy HTML emails.
- How to A/B test SMS campaigns
Learn how to A/B test SMS campaigns and what aspects to test in this article.
- Understanding statistical significance in Klaviyo campaigns
Learn how to tell whether or not an A/B test is considered statically significant or not. For instance, if a campaign variation A has an open rate of 15% and variation B has an open rate of 14%, how do you know for sure if variation A or B performed better?
- Understanding statistical significance in flows
Learn when an A/B test’s results are considered to be statistically significant in Klaviyo flows. This article will explain statistical significance for flows and when it is decided.