Learn how to strengthen your AI Service Agent’s ability to resolve shopper inquiries by testing its responses and adding knowledge for any gaps you discover.
AI Service Agent is a part of Klaviyo Service and is currently in public beta. While this product remains in beta, functionality is not complete, and will be subject to change frequently throughout the beta period. By enabling AI Service Agent through the onboarding workflow, you agree to the beta terms and conditions as provided in your contract with Klaviyo. While this feature is free during the beta period, it will be sold separately upon launch to general availability.
If you do not have access to the AI Agent tab in Klaviyo’s main navigation yet, join the AI Agent beta. Note that you must be on a paid Klaviyo email plan to be approved. You must also have Customer Hub enabled to join the AI Service Agent beta. Participation in this beta is currently available in English for those who operate a Shopify Online Store.
For feedback about AI Service Agent functionality, email customerhub@klaviyo.com.
Before You Begin
This guide provides guidance for training your AI Service Agent, which is currently in a beta phase. Before proceeding, confirm that the AI Service Agent is enabled for your account.
Testing your AI Service AgentTesting your AI Service Agent
While your AI Service Agent automatically syncs information from your website to help answer shopper questions, manual testing is essential. Testing ensures the AI Service Agent delivers accurate responses and minimizes unnecessary handoffs to human agents.
“Testing” means using the preview environment, the Setup tab in Klaviyo, to interact with your AI Service Agent as if you were a customer. This allows you to:
- Get familiar with what the AI Service Agent already knows
- Identify knowledge gaps where it needs more information
- Avoid duplicating information by uploading information that the AI Service Agent already understands
Before launching your AI Service Agent, thoroughly test it with a wide range of customer questions to catch any gaps. After launch, continue testing at least every 30 days, and more often if your catalog is large or frequently updated. Always test when you add new products, update your site, or change policies. The bigger and more complex your website, the more frequent and thorough your testing should be.
What to testWhat to test
Ask your AI Service Agent a variety of questions in the preview environment, such as:
- Product specific questions (“How do I care for it?” or “What material is it made of?”)
- Product recommendations (“Which shirt pairs with this?”)
- Order status updates (“Where is my order?”)
- Company policies (“What is your return policy?”)
- Common questions (“How can I contact support?”)
How to test
If your AI Service Agent is live and you’d prefer to turn it off before testing or training, click the Turn off button in the top right corner your AI Agent settings in Klaviyo. Inbound messages are redirected to the Inbox where they’ll need to be manually replied to until the AI Service Agent is reactivated.
To test your AI Agent:
- In your Klaviyo account, select Conversations >AI Agent.
- Select the Setup tab. By default, the page selector is unselected, so the AI Service Agent uses its knowledge of your entire site to respond.
- If you want to test the AI Service Agent’s ability to answer product-specific questions, use the page selector to choose a specific product page before asking your question. This simulates a shopper viewing that page and provides the AI with additional context to give more precise answers.
- Enter a test question in the chat interface, as if you were a shopper.
- Review the AI Service Agent’s response to evaluate its accuracy and helpfulness.
- Tip: Try follow-up questions to probe the AI Service Agent’s knowledge further.
- Tip: Try follow-up questions to probe the AI Service Agent’s knowledge further.
- Watch for escalation; if it begins the process of handing you off to a human support team member, note this as a knowledge gap to train your AI Service Agent on. Handoff attempts from the preview environment in Klaviyo do not actually trigger an agent handoff to happen (i.e., a ticket is not actually created in your connected helpdesk).
- Repeat this process as needed, testing different topics or support scenarios your shoppers may encounter.
As you test, you can either keep a list of knowledge gaps to address later or immediately add knowledge for questions the AI Service Agent can’t answer.
Add knowledge to your AI Service AgentAdd knowledge to your AI Service Agent
If the AI Service Agent can’t answer a specific question, you can manually provide the information it needs to train on the topic. Gaps may stem from:
- Deeply nested web pages or dynamic content that weren’t automatically ingested
- Information not directly available on your site
To add knowledge:
- In your AI Agent settings, select the Knowledge tab.
- Click Add new and select the method for uploading information:
- Snippets: Input concise answers for common asked questions. Tip: include the question or title within the provided answer field to improve the AI Service Agent's ability to retrieve and use the information.
- Files: Upload documents like user guides or product details.
-
Webpages: Provide URLs for the AI Service Agent to extract content from.
- Click Save to apply the changes.
After adding new information, return to the Setup page and ask the same question to confirm the AI Service Agent can now answer it correctly. Continue this process for each knowledge gap you identify.
When you’re confident in the AI Service Agent’s responses and knowledge coverage, click Turn on to turn your AI Service Agent live (if you haven’t already).
Next steps and ongoing maintenance
After setting your AI Service Agent live, regularly monitor its performance using the AI Agent overview dashboard. Review new or recurring customer questions and retest the AI Service Agent to maintain response quality.
To keep your AI Service Agent’s knowledge current:
- Remove outdated knowledge: In the Knowledge tab, click the 3 dots next to any source you want to remove, then select Delete.
- Resync webpages: If you’ve updated a webpage, but notice your AI Service Agent’s answers aren’t reflecting the new information, click the 3 dots next to that source and select Resync.