Getting started with global holdout groups

You will learn

Learn how to use global holdout groups to understand the impact and efficacy of your marketing. Global holdout groups provide a way to exclude a number of profiles from your messaging to determine the incremental success of your marketing program. These holdout groups allow you to compare and then analyze the base value of your marketing programs, as well as ways to continually optimize these strategies.

Before you begin

You must have at least 400,000 profiles (note that this is total profile count) to qualify for holdout groups. This ensures that Klaviyo has enough profile data to compare the holdout group to your other profiles receiving marketing. We additionally recommend that you run holdout groups for 3 months to gather enough data.

Learn more about when to or when not to use holdout groups.

Using holdout groups appropriately

How to use holdout groups

Holdout groups should be used when you want to test the efficacy of your marketing programs, and potentially any changes that you may make to these strategies. Holdout groups provide a comparison by excluding profiles from marketing messages, except for transactional flow messages and any campaigns or flows you override.

For example, you want to consider using holdout groups if you want to establish:

• The clear value and impact of your overall marketing program against costs.
• When making changes to your marketing program (e.g., adding in a new channel), the value that is associated with these incremental changes.
• If certain marketing activities are actually detrimental and producing negative results and need to be adjusted (e.g., too many flows at the wrong times to customers).

Additional factors to consider with holdout groups

It’s important to consider the timing and application of when you may use holdout groups.

Since those in a holdout group are withheld from receiving marketing emails, SMS, and push notifications sent via campaigns or flows, except for transactional flow messages and any overrides, consider the following factors:

• You can only have 1 active holdout group at a time to use for comparison with your marketing programs.
• A holdout group applies to all channels used in a campaign or flow; it cannot be turned on for one specific channel vs. another.
• Consider whether the timing of your holdout test could interfere with important seasonal ecommerce events or times of year (e.g., Black Friday Cyber Monday).
• Which messages profiles in holdout groups should still receive. Learn more about overriding messages.

Holdout groups vs. A/B testing

Holdout groups are different in nature than A/B testing and the 2 functions should not be confused. Holdout groups allow you to understand the impact of your marketing communication by withholding marketing messages from a group of people, except for transactional flow messages and any campaigns or flows you override. In contrast, A/B testing allows you to compare 2 or more versions of your marketing communication. For example, you can compare 2 versions of an email campaign.

Creating a holdout group

1. To create a holdout group, head to Analytics > Experiments and click on the Global Experiments tab.

2. Click on Create test.

You will then be brought through a series of steps to set up and customize your holdout group.

3. Use the field on the left to enter the percentage of marketable profiles that you want to put in your holdout group. Enter a number greater than 0 and less than 100, with up to 2 decimal places. As you update the percentage, the estimated starting holdout size on the right updates.

4. Once you are satisfied with your percentage, click Continue.

5. Click the Select date field and choose your end date, then click Continue. Your holdout group starts today when you finish setup and click Start test.

You can also ensure that certain flows deemed necessary are still received by profiles in your holdout group. To do this, you will override the holdout group rules for certain flow(s). For example, many welcome flows may be important enough that they should still be received by your holdout customers and therefore overridden.

6. To make sure your profiles still receive certain flow(s), search by the flow name in the Search flows field to the left. You can also narrow your search for a particular flow by opening the Select tags dropdown and selecting certain tag(s) from the dropdown.

You can additionally narrow your search for a certain flow by opening the All dropdown and choosing a flow type, such as Welcome Series or Browse Abandonment. By choosing an option, you will only see that type of flow listed below.

Finally, you can choose to look for flows based on status (Live, Manual, Draft). Open the All statuses menu and choose one of the options; this will then narrow your list below.

7. Once you have found the flow(s) you want to override (i.e., your profiles will receive these flows), select the checkbox next to each applicable one and click Review.

If you do not wish to select any to override, click Skip.

8. In the Details tab, review the settings including start and end dates and percentage of marketable profiles in the holdout group.

9. Click on the Overridden flows tab to review any flows that will still send to your holdout group profiles, as shown in the example below.

10. If you have any changes to make click Back. Otherwise click Start test if you are ready for your holdout group to be active.

Editing or ending your holdout groups

If you have set up a holdout group and want to edit certain settings (i.e., name or end date) or end the group test early, head to Analytics > Experiments > Global Experiments.

Here you will find the global holdout card (it will be the card with data above the history card).

Editing an active holdout group

1. If you want to edit the name of your holdout group or the end date, click on the 3 dot menu in the upper right corner of the card.

2. Choose Edit details from the menu.

3. From here, you can edit your holdout group’s name in the Edit test name field. You can also adjust the end date by clicking on the Change test end date field and choosing a new date from the calendar widget.

4. Once you are done with your edits, click Save.

Ending an active holdout group test

If you find that you wish to end your test early, you may do so via this card as well.

1. Click on the 3 dot menu in the upper right corner of the card.

2. Choose End test from the menu.

3. A modal will then appear asking you to confirm if you want to end the test and have your holdout group start receiving all marketing messages again. To confirm, click End test. Otherwise, click Cancel to avoid completing this action.

Reviewing overridden flows

1. If you want to review what flow(s) you may have overridden for your holdout group, click on the 3 dot menu in the upper right corner of this card.

2. Choose Active overrides from the menu.

3. Review the flow override(s) that you have chosen for this particular holdout group, if applicable.

Overriding additional flows

After you have gone through the initial holdout group setup, you may find that you want to add more overridden flows. In other words, you may have other flows that you want your holdout group to receive.

1. To add more overridden flows, navigate to Flows. Find and click on an existing flow from the list or click on Create flow to override holdout groups.
2. Once inside the flow, either create or click on the trigger in your flow.

The override option appears in the trigger’s Settings section while a global holdout test is active.

3. Under Settings, select Override global holdout group.

The change saves automatically. A confirmation message tells you that holdout group participants will receive communications from this flow.

To stop overriding the holdout group for this flow, clear Override global holdout group. This change also saves automatically.

Overriding individual campaigns (email, SMS, or push)

It may be necessary to allow certain campaigns to reach your holdout groups. For example, you may want your global holdout groups to receive an individual welcome email from your brand. In this case, you will need to override the specific campaign to ensure that your global holdout groups receive this particular message or campaign.

1. Navigate to Campaigns.
2. Search in the field or find the appropriate draft campaign from the list below. If you are creating a new campaign you can also add the override directly during setup at the same time.

3. In the Recipients step, expand Global holdout group. This option appears while a global holdout test is active.

4. Turn on Override global holdout group to allow this campaign to send to eligible profiles in the holdout group. Leave it off to keep those profiles excluded.

5. Click Next to save your recipient settings and continue.

6. Finish setting up your message, then click Next to open Review.

On Review, confirm that the campaign will send to the active global holdout group before scheduling it.

Adding new marketing channels at a later date

If you add a new channel to an existing campaign (e.g., SMS), your settings will be respected from your initial campaign setup, including if you have chosen to override global holdout groups.

If you wish to adjust this, follow the setup directions above and instead toggle the override option off.

Reviewing if a profile is or was part of a holdout group

You can review if a particular profile(s) is currently or was part of a global holdout group at one point.

Within a singular profile

1. Navigate to Audience > Profiles.
2. Search in the field above or choose a profile from the list below. A banner, as shown in the example below, will let you know if a profile is part of global holdout group.

You can also see if a profile was previously part of a holdout group and when this occurred. Removal from a holdout group means that the test was ended and the profile was removed from it.

Within flows

1. Navigate to Flows.
2. Use the search field above or find the flow from the list below.
3. Click on a message in the flow that has been skipped.
4. In the message’s settings panel, click View details in the Performance section.

5. Open Recipient activity, then select Skipped. This section shows why profiles were skipped at send time for the selected flow message, including membership in a global holdout group.

Within campaigns

Skipped-recipient data is available only for campaigns sent on or after April 15, 2026.

1. Navigate to Campaigns.
2. Search in the field or find the appropriate sent campaign from the list below.
3. Open Recipient activity, then select Skipped. This section shows why profiles were skipped for this campaign message, including membership in a global holdout group.

Reviewing analytics on holdout groups

Latest global holdout group card

The global holdout card at the top of the analytics section shows your latest holdout test, whether it is active or completed. This card helps you review your marketing strategies using conversion rates, marketing lift, and win probability. Use the conversion metric selector above the card to choose the metric you want to compare.

The top of your card shows the overall marketing lift percentage. This is the relative difference between the treatment group’s conversion rate and the holdout group’s conversion rate: (treatment rate − holdout rate) ÷ holdout rate, displayed as a percentage. If the holdout conversion rate is 0, marketing lift displays 0%. It helps you understand the overall impact of your marketing program, taking into account any campaigns or flows you chose to override.

Below marketing lift, you will see data on:

• Holdout percentage
  The percentage selected for your holdout group during setup. Profiles in the group are excluded from marketing messages except for transactional flow messages and any campaigns or flows you override.
• Win probability
  Probability that your marketing program provides a meaningful improvement over the holdout group.

• Treatment group
  The main group receiving all of your marketing.
• Holdout group
  The group excluded from marketing emails, SMS, and push notifications sent through campaigns and flows, except for transactional flow messages and any campaigns or flows you override.
• Marketable profiles
  Profiles reachable through at least 1 of email, SMS, or push are considered marketable for holdout reporting. For email, there are 2 main categories of marketable profiles:
  • Subscribers
    Subscribers have filled out a sign-up form or otherwise explicitly consented to receive email marketing.
  • Profiles added by general engagement
    It is possible for someone to share their email with you without explicitly consenting to ongoing email marketing. For example, someone who placed an order or abandoned a checkout on your site may have added their email address during the checkout process, but never explicitly opted in.

The last 2 columns show the number of unique profiles who performed the selected conversion event and their percentage of the group’s marketable profiles during the test. A profile that converts more than once is counted once. These results include conversions whether or not they are attributed to a Klaviyo message.

History card

The history summary card provides the data for all your holdout groups. This card is especially helpful for seeing any incremental changes that you may have made to your marketing programs and how these affected the conversions and uplift. Here, you can trace back these changes or points in time and track what was successful in your programs.

Columns will represent the following:

• Name
  Name of the global holdout group you set up.
• Start date
  When the global holdout group was set up and first active.
• End date
  When the global holdout group ended and profiles were removed from it.
• Holdout size
  The percentage of profiles out of your total marketable profiles that are represented in the holdout group. This is derived from your initial holdout group setup.
• Marketing lift
  The percentage lift of your marketing program compared to those in the holdout group using your account’s selected conversion metric. This shows the overall success of your marketing strategies compared to those not receiving all or most of your marketing.
• Win probability
  Probability that your marketing program provides a meaningful improvement over the holdout group.

You can also click the arrows next to any of these columns to sort by names, dates, holdout sizes, marketing lift, and win probability by ascending or descending order.

By clicking on any of the global holdout names, you will be brought to an individual analytics card with data on this particular group compared to your marketing programs.

Additional resources