How to test your Customer Agent

You will learn

How to test Customer Agent in the Preview sidebar, debug responses you don’t expect, and validate skills before shoppers see them.

Before you begin

You’ll need:

• Customer Agent enabled
• Some content, Guidance, and skills configured (you can iterate, but testing without any setup won’t tell you much)
• Permission to edit Customer Agent if you want to change settings or rerun simulations.

The preview uses live tools and can update Klaviyo profiles or connected systems. Use a test profile and test data for actions you don’t want applied to a shopper’s account.

Why testing matters

Customer Agent’s behavior comes from a stack of signals — your content, Guidance, skills, tools. A small change in any of those (rewriting your brand summary, adding a communication style rule, attaching a new tool) can ripple into responses you didn’t expect.

Test before launching, and re-test every time you make a meaningful change. Plan a thorough test pass before going live, then check in every 30 days, or any time you update content for a sale, a new product line, or a policy change.

Set it up

1. Open the Preview sidebar

1. Navigate to Customer Agent > Guidance.
2. Click Preview if the sidebar is closed. If the sidebar has both Build and Preview tabs, select Preview.

The sidebar tests your content, skills, tools, and Guidance. It includes draft skill changes. On the Guidance page, it also uses unsaved changes to tone, communication style rules, and escalation rules, so its responses can differ from what shoppers currently receive.

Before sending your first message, choose the context for your test:

• Web page: For web chat, select a page from your indexed storefront, such as a product or collection page, to test questions about what the shopper is viewing.
• Channel: Open the channel menu, initially labeled Web chat, and choose the channel you want to test, such as Web chat, Text message, Email, or WhatsApp.
• Profile: Select a test profile to test with its order history, subscriptions, and other available customer data. Select Signed out profile to test without signing in as a customer.

2. Send sample questions

Enter a shopper-style message in Ask a question..., then click Submit. A starter set:

• “How do I care for it?” (tests product question handling)
• “What material is this made of?” (tests product knowledge from your catalog)
• “Where is my order?” (tests the Order tracking skill)
• “What’s your return policy?” (tests content retrieval)
• “Can I talk to a person?” (tests handoff)

Mix in questions specific to your brand — anything you’d expect a real shopper to ask.

Keep related follow-up questions in the same conversation. To start a separate test or change the channel or profile, click Reset, then choose your test context and send another message. Reset clears the test conversation.

3. Review what happened under the hood

After a response, expand the action summary in the conversation to inspect what Customer Agent did. Open a tool or source entry to see its details. The available entries show:

• Which skill Customer Agent picked
• Which tools ran and what they returned
• Which content was retrieved
• The final response and any escalation rules that matched

Use these details to identify whether you need to update skill instructions, content, tools, or Guidance.

Testing skills

If you’ve built a fully custom skill, navigate to Customer Agent > Skills, open the skill, and use its Preview sidebar. Focus your testing on 3 things:

• Router behavior — Send messages that should match your skill’s “When to use this skill.” Confirm your skill was actually picked.
• Response behavior — Walk through the sections in your skill’s “How to respond.” Make sure each section’s behavior happens — including tool usage and any handoff actions.
• Tool usage — If your skill uses tools, confirm they fire at the right step with the right parameters.

Iterate on “When to use this skill” if the router is picking it wrong, and on “How to respond” if the behavior is off.

Retest after editing the skill. When you’re ready for shoppers to use the changes, click Publish for a draft skill or Update for a live skill with pending edits. Testing a draft does not publish it.

Debugging a bad response

When Customer Agent gives a response that doesn’t seem right, work through it in this order:

1. Was the right skill picked? For a fully custom skill, review “When to use this skill” and check whether it clearly describes the requests the skill should handle.
2. Was the right content retrieved? If Customer Agent used outdated or wrong content, fix the source — see How to add and manage content.
3. Did tools run correctly? Check the tool calls for missing parameters or unexpected returns.
4. Did Guidance shape the response correctly? Tone, communication style rules, and escalation rules all layer on top of skill output.

Fix the relevant skill, content, tool, or Guidance setting, then start a new test to confirm the result. On the Guidance page, click Save when you’re ready to apply tone or rule changes to shopper conversations.

Troubleshooting

Symptom: Customer Agent picks the wrong skill.
Likely cause: Skill descriptions overlap, or “When to use this skill” is too broad or too narrow on a fully custom skill.
Fix: Tighten the skill’s “When to use this skill” content. Test multiple phrasings of the same intent.

Symptom: Customer Agent doesn’t use a tool you expect.
Likely cause: Tool isn’t attached to the skill, or “How to respond” doesn’t reference the tool at the right step.
Fix: Confirm the tool is attached. In “How to respond,” insert the tool inline at the step where it should run.

Symptom: Customer Agent uses the wrong content.
Likely cause: Conflicting sources, or another source matches the shopper’s question more closely than the source you expected.
Fix: Audit and consolidate conflicting content. Re-sync sources you’ve recently edited.

Symptom: Response tone or style is off.
Likely cause: Tone of voice or communication style rules need adjustment.
Fix: Review Guidance settings. Iterate on tone or add a specific communication style rule.

Review simulations

Simulations run Customer Agent through shopper scenarios and grade the responses against expected behavior. Rerun uses published skills and saved Guidance. Use the Preview sidebar to test draft skills or unsaved Guidance first. To review existing simulations:

1. Navigate to Customer Agent > Train.
2. Open a scenario and review its simulated conversation and results.
3. After saving or publishing your changes, click Rerun to test the scenario again. Wait for the run to finish, then review the updated results.

Rerun requires permission to edit Customer Agent and is unavailable while auto-training or the scenario’s current simulation is running.

Additional resources