Best practices for A/B testing reference

You will learn

Learn best practices for using Klaviyo's A/B testing features to help optimize your communication with customers.

Data-driven decisions are the difference between good and great marketing. When it comes to knowing what your audience will respond well to, A/B testing is your closest ally. An A/B test allows you to show different copywriting and creative work to a subset of your audience to test what makes them more likely to complete the call-to-action (CTA) of your form or email. To make sure you aren’t leaving money on the table, let’s dive into six best practices for your A/B tests.

Develop a hypothesis

You can consider how to develop a hypothesis in two different ways — on a micro and macro scale. If you feel as if your marketing strategy is going well and you want to fine-tune your communication with your customers, you can consider changing some of the following:

• Colors for CTA
• Copy on CTA
• Subject line
• From name
• Offer amount
• Imagery using predictive analytics

If you’re considering switching up your marketing in a major way, you can use A/B testing to see if your new idea resonates with your audience on a macro scale. Examples of this might be:

• Different brand colors
• New layouts
• Change in brand voice
• Participation with a new brand community

Test one variable at a time

While you may have a handful of ideas that you want to test, it’s best to test one variable at a time. For instance, if you edit the color of the CTA and also the text on it and see a spike in form submit rates, you won’t know if the spike was due to the change in color or the copy on the CTA. By testing one variable at a time, you can be sure you know what your audience responds to the most.

Get a large enough audience

Imagine you have a great idea for a form, you A/B test it, and, on the first day, the first 10 people who see the form click through and make a purchase. Over the course of a week, the next 2000 people exit out of the form immediately. If you had ended the test after the first day, you would think that your form was great, even though a larger audience showed otherwise. By waiting until your results are statistically significant or until you have a good sample size of viewers, you’ll ensure that you know which form is truly better for your brand.

No more than 4 variations

In the same vein, while still testing one variable at a time, the more variations you add, the bigger the audience you’ll need to make sure you get reliable results. To get your results in a timely manner so you can perform more tests, use no more than 4 variations of one variable per test.

Don’t edit a live test

After you’ve done the hard work of putting your test together, refrain from making tweaks to your test that can impact the results. Luckily, in Klaviyo, you cannot make changes to a live A/B test for campaigns and forms, so this best practice applies to flows specifically. If you find yourself wanting to make a change to a template, end your A/B test first or, in the case of a campaign, cancel the email send. Then, restart the test to ensure you’re getting the most accurate data.

Ask for feedback

If you’re planning on making a big shift in your marketing, A/B testing can be a great place to see if your audience responds well to new ideas. Asking for feedback directly from your customers is a useful strategy to use in tandem with your A/B tests. An easy way to communicate with customers who are a part of an A/B test is to tag those that interact with your CTA.

After you’ve tagged those that have interacted with your form or email, you can follow up with a plain text email asking for their feedback. Because you will only be gathering information from those who interacted with your form or email, you’ll want to also pull in customers who are very engaged with your brand. Feedback is a great way to engage your community and make them advocates for your brand.

Additional resources

• How to A/B test campaign emails
• How to A/B test flow branches
• A/B testing a signup form