Best practices for A/B testing reference
You will learn
Learn best practices for using Klaviyo's A/B testing features to help optimize your communication with customers.
An A/B test allows you to compare different copywriting and creative work with subsets of your audience to learn what makes them more likely to complete the call to action (CTA) of your form or email. Use these 6 best practices to plan your tests and interpret the results.
Develop a hypothesis
Before building your test, write down what you expect to improve and why. For example, you might predict that more specific CTA copy will increase form submissions because visitors will know what they are signing up for.
You can consider how to develop a hypothesis in 2 different ways: on a micro and macro scale. If you feel as if your marketing strategy is going well and you want to fine-tune your communication with your customers, you can consider changing some of the following:
- Colors for CTA
- Copy on CTA
- Subject line
- From name
- Offer amount
- Imagery
If you’re considering switching up your marketing in a major way, you can use A/B testing to see if your new idea resonates with your audience on a macro scale. Examples of this might be:
- Different brand colors
- New layouts
- Change in brand voice
- Participation with a new brand community
Test 1 variable at a time
While you may have a handful of ideas that you want to test, it’s best to test 1 variable at a time. For instance, if you edit the color of the CTA and also the text on it and see a spike in form submit rates, you won’t know if the spike was due to the change in color or the copy on the CTA. Testing 1 variable at a time helps you connect a difference in results to the change you made.
Get a large enough audience
Imagine you have a great idea for a form, you A/B test it, and, on the first day, the first 10 people who see the form click through and make a purchase. Over the course of a week, the next 2,000 people exit out of the form immediately. If you had ended the test after the first day, you would think that your form was great, even though a larger audience showed otherwise. Allow enough time to collect results from a larger audience, then check whether your form’s A/B test results are statistically significant. A large sample alone does not establish that 1 variation is better.
No more than 4 variations
In the same vein, while still testing 1 variable at a time, the more variations you add, the bigger the audience you’ll need to make sure you get reliable results. To get your results in a timely manner so you can perform more tests, use no more than 4 variations of 1 variable per test.
Don’t edit a live test
After you’ve done the hard work of putting your test together, refrain from making tweaks to your test that can impact the results. If you need to change content, start a new test so results from before and after the change stay separate.
- For sign-up forms, you can change test settings while a test is live, but you cannot change the live variations’ content. Editing content creates a new draft test. Your current test remains live until you publish the new test.
- For flow emails, when prompted after editing a live test, choose End and publish new test and then Continue. Continuing the current test can skew its results.
- For a campaign that is already sending, cancel the remaining send if you need to change the content. Cancellation does not recall delivered emails. Clone the canceled campaign or create a new campaign, review its recipients, and set up a new test.
Ask for feedback
If you’re planning on making a big shift in your marketing, A/B testing can be a great place to see if your audience responds well to new ideas. Asking for feedback directly from your customers is a useful strategy to use in tandem with your A/B tests. For email tests, you can add a profile property when a recipient clicks your CTA. Create a segment using that property to identify people to ask for feedback.
For sign-up forms, create a segment of subscribers who joined through the form to identify people to ask for feedback.
After you’ve identified these subscribers, you can follow up with a plain text email asking for their feedback. Because you will only be gathering information from those who interacted with your form or email, you’ll want to also pull in customers who are very engaged with your brand. Feedback is a great way to engage your community and make them advocates for your brand.