How to optimize a sign-up form's display settings

You will learn

Learn how to run an optimization test on a Klaviyo sign-up form to determine the best time for the form to display to website shoppers. In an optimization test, Klaviyo runs a series of AI-generated experiments to determine the most effective display timing settings that increase a form’s submission rate.

An optimization test is similar to an A/B test in that its purpose is to identify areas that work well or need improvement. However, an optimization test uses AI to choose display timing variations and run successive test phases automatically, saving you time.

This content requires Functional cookies to be enabled.

Update Cookie Settings

Before you begin

Create an optimization test

Create and run an optimization test with your desired test settings to determine the optimal display timing settings for your sign-up form. When the test finishes, the winning version is live on your site. If the control wins or the results are inconclusive, your original display timing settings stay in place.

Note that you can only run 1 optimization test per sign-up form at a time.

To create an optimization test:

1. Select Sign-up forms from the left-hand navigation.
2. Create a new sign-up form, or open the form editor for an existing form by selecting the 3 dots, then Edit.
   

1. In the editor’s left sidebar, select the Create A/B test beaker icon, then select Create A/B test in the panel that opens.

2. Select Optimization test. You cannot begin an optimization test if an A/B test is currently in progress.
   

3. Under Test name, you can change the default name of your test. Beneath this, verify the timing settings next to Control. The control is your current form.

If the form has low traffic, a warning explains that the test may take up to 90 days. Consider using a higher-traffic form to collect enough data sooner.

1. Select Continue to test settings.
2. On the General tab, enter the percentage of traffic for the Test group. The remaining traffic goes to the Control group:
   • Control group
     This is the group that will always see the current version of the form.
   • Test group
     This is the group that’s tested throughout the phases of the optimization test.
     

3. Optional: Select the Advanced settings tab to choose which display rules AI can use. You can allow time delay, scroll percentage, and exit intent variations, and set a minimum time delay or scroll percentage. Keep Allow time delay variations, Allow scroll percentage variations, or both selected to start the test.

4. When you’re ready to publish the test on your site, select Start optimization.

The test starts with a learning phase that compares display timing variations with your current form. Klaviyo uses the results to choose variations for subsequent iteration phases. A test can finish when the same settings are selected for 3 consecutive iteration phases, after a final comparison when further improvement is unlikely, or when variations repeatedly perform worse than the control. Tests run for up to 90 days. When the test ends, the winning settings go live automatically; inconclusive results keep the control.

Monitor an ongoing optimization test

To monitor an optimization test while it’s running:

1. Select Sign-up forms from the main navigation.
2. Click on the form that has an optimization test in progress. It will feature a small “Running an optimization” banner in the Status column.
3. In the menu bar, click on the A/B test results tab.
4. Make sure that the dropdown at the top is set to Optimization test, along with the name of the test you’d like to view.
   

From here, you’ll see an overview of the ongoing optimization test, including a timeline of its learning and iteration phases and the following results for the phase in progress:

• Submit rate lift
  The percentage difference between the control group’s submit rate and the highest performing variation’s submit rate.
• Win probability
  The percentage likelihood that the variation will perform better than the control group.
• Current winning variation
  The display timing settings of the highest-performing test variation in the current phase. The test-group traffic percentage appears beneath the test name.

On the bottom half of the A/B test results tab, a results table displays each phase in the optimization test, including the phase in progress. Each phase is broken down by:

• Start date
  The date that this phase began.
• Eligible unique views
  The number of unique users eligible to view a sign-up form in this testing phase. Because you’re testing display timing, views will be counted by the number of unique users who could have seen a form in order to accurately compare form variations.
• Submit rate
  The number of unique submitted form events divided by the number of unique users eligible to see the form variation (Eligible unique views). For example, if 2 unique users are eligible to see a form variation and 1 of them submits it, the submit rate will reflect 50%.
• Win probability
  The win probability for the variant being tested against the control group.
• Submit rate lift
  The percentage increase of submission rates in comparison to the control group.

Click the dropdown arrow on any phase to see the key details and data for the control group, as well as a breakdown of the raw data (i.e., Submit rate, Win probability, and Submit rate lift) for each variant tested during the phase.

You can view the data from each phase while the optimization test is running and after it ends.

Edit an optimization test name

To edit the name of an optimization test while it’s in progress:

1. On the Sign-up forms page, select the form with the optimization test running.
2. Click the A/B test results tab, then select Optimization test and the name of the test you want to rename.
3. In the top right corner, select the 3 dots, then Edit details.
   

4. Under Test name, replace the current name with your desired change.
5. Click Save.
   

Edit a form during an optimization test

You cannot make edits to the sign-up form itself while an optimization test is in progress. If you enter the editor for a sign-up form with an ongoing test, a modal will ask if you want to end the test in order to make edits to the form, and how you’d like to end it. You can either:

• Keep control variation
  Keep the original form and its display timing settings. This choice does not mean the control performed best in the test.
• Replace control with highest performing variation
  Replace the original form’s display timing settings with those of the highest-performing test variation from the current phase. This variation may still perform worse than the control, so check its submit rate lift before selecting it.

Select End test to confirm your choice and open the form editor. If you end an ongoing test to modify the sign-up form, you’ll need to create a new optimization test to start again.

Manually end an optimization test

To change the traffic distribution or allowed display timing variations, end the current optimization test and create a new one with the desired settings. To end the test:

1. On the Sign-up forms page, select the form with the optimization test running.
2. Click the A/B test results tab, then select Optimization test and the name of the test you want to end.
3. In the top right corner, select the 3 dots, then End test.
   

4. In the confirmation dialog, select Keep control variation to retain the original timing settings, or Replace control with highest performing variation to use the highest-performing test variation from the current phase. Check the submit rate lift before replacing the control.

1. Select End test.

How a winning variation is chosen

When an optimization test ends automatically, the sign-up form updates to the winning display timing settings, or stays the same if the control is selected. The test status changes from Live to Completed on the A/B test results page.

For an automatically completed test, an overview of the test and finalized results will display for your analysis, and 1 of 3 outcomes will show:

1. Successfully optimized: A winning variation was identified
   
   • The AI test identified a variation that performed better than the control group. A description of the winner will display under Winning variation.
2. Successfully optimized: The control is the winner
   
   • Klaviyo selected the control as the winner, so your form keeps its original display timing settings. These settings appear under Winning variation - Control. The Submit rate lift card shows a dash because the control is the baseline for comparison.
3. Inconclusive results:
   • The test did not reach a conclusive result. This can happen when it reaches the 90-day limit or when a final comparison does not confirm the predicted winner. The control is kept as the winning variation, and the Submit rate lift card shows a dash.

Next steps

See how to view sign-up form responses to dive deeper into the information collected from your new subscribers in your sign-up forms.

If you have multiple sign-up forms live on your website, consider enabling collision prevention display settings to keep multiple sign-up forms from appearing at once.

Additional resources