How to test and train your Customer Agent

Learn how to strengthen your Customer Agent’s ability to resolve shopper inquiries by testing its responses and adding knowledge for any gaps you discover.

Before You Begin

This guide provides guidance for training your Customer Agent, which is currently in a beta phase. Before proceeding, confirm that Customer Agent is enabled for your account.

Testing your Customer Agent

While your Customer Agent automatically syncs information from your website to help answer shopper questions, manual testing is essential. Testing ensures Customer Agent delivers accurate responses and minimizes unnecessary handoffs to human agents.

“Testing” means using the preview environment, the Playground tab in Klaviyo, to interact with your Customer Agent as if you were a customer. This allows you to:

• Get familiar with what Customer Agent already knows
• Identify knowledge gaps where it needs more information
• Avoid duplicating information by uploading information that Customer Agent already understands

Before launching your Customer Agent, thoroughly test it with a wide range of customer questions to catch any gaps. After launch, continue testing at least every 30 days, and more often if your catalog is large or frequently updated. Always test when you add new products, update your site, or change policies. The bigger and more complex your website, the more frequent and thorough your testing should be.

How to test

To test your Customer Agent: 

1. In your Klaviyo account, select Service > Customer Agent.
   
2. Select the Playground tab. By default, the page selector is unselected, so Customer Agent uses its knowledge of your entire site to respond.
   • If you want to test Customer Agent’s ability to answer product-specific questions, use the page selector to choose a specific product page before asking your question. This simulates a shopper viewing that page and provides the AI with additional context to give more precise answers.
3. Enter a test question in the chat interface, as if you were a shopper.
4. Review Customer Agent’s response to evaluate its accuracy and helpfulness.
   • Tip: Try follow-up questions to probe Customer Agent’s knowledge further.
     
5. Watch for escalation; if it begins the process of handing you off to a human support team member, note this as a knowledge gap to train your Customer Agent on. Handoff attempts from the preview environment in Klaviyo do not actually trigger an agent handoff to happen (i.e., a ticket is not actually created in your connected helpdesk).
   
6. Repeat this process as needed, testing different topics or support scenarios your shoppers may encounter.

As you test, you can either keep a list of knowledge gaps to address later or immediately add knowledge for questions Customer Agent can’t answer. 

Add knowledge to your Customer Agent

If Customer Agent can’t answer a specific question, you can manually provide the information it needs to train on the topic. Gaps may stem from:

• Deeply nested web pages or dynamic content that weren’t automatically ingested
• Information not directly available on your site

To add knowledge:

1. In Customer Agent, select the Knowledge tab.
    
2. Click Add new and select the method for uploading information:
   • Snippets: Input concise answers for common asked questions. Tip: include the question or title within the provided answer field to improve Customer Agent's ability to retrieve and use the information.
   • Files: Upload documents like user guides or product details.
   • Webpages: Provide URLs for Customer Agent to extract content from.
     
3. Click Save to apply the changes.

After adding new information, return to the Playground page and ask the same question to confirm Customer Agent can now answer it correctly. Continue this process for each knowledge gap you identify.

When you’re confident in Customer Agent’s responses and knowledge coverage, turn on at least one channel in the panel at bottom left corner of your screen.

Next steps and ongoing maintenance 

After setting your Customer Agent live, regularly monitor its performance using the overview dashboard. Review new or recurring customer questions and retest Customer Agent to maintain response quality.

To keep your Customer Agent’s knowledge current:

• Remove outdated knowledge: In the Knowledge tab, click the 3 dots next to any source you want to remove, then select Delete.
• Resync webpages: If you’ve updated a webpage, but notice your Customer Agent’s answers aren’t reflecting the new information, click the 3 dots next to that source and select Resync.

Additional resources

• What types of questions can Customer Agent answer
• How to configure handoff settings for the Customer Agent
• Getting started with Customer Agent