Playground

The Agent Playground

The Playground is your private testing environment for interacting with your AI agent exactly as your end-users will. Every change you make to your agent's sources, system prompt, or model configuration can be validated here instantly, without affecting live conversations. Think of it as a staging environment for your AI -- a place where you can experiment freely, catch issues early, and build confidence before going live.

No matter how well you have trained your agent, deploying without thorough testing is a risk. The Playground eliminates that risk by giving you a real-time, fully functional replica of the production chat experience.

Why the Playground Matters

Deploying an untested AI agent to customers is like shipping code without running your test suite. The Playground exists to prevent three critical failures:

• Inaccurate responses that erode customer trust and create support tickets instead of resolving them.
• Tone mismatches where the agent sounds too casual, too robotic, or off-brand for your audience.
• Knowledge gaps where the agent fails to answer questions your customers frequently ask.

By investing time in the Playground before deployment, you dramatically reduce the risk of poor first impressions and ensure your agent delivers value from day one.

Interface Walkthrough

When you open the Playground, you will see a chat interface that mirrors the production widget your users interact with.

Getting Started with the Playground

Evaluating Response Quality

Every response your agent generates should be evaluated across four dimensions:

Accuracy

Does the response contain factually correct information? Cross-reference the agent's answer against your actual business data. If the agent says your return window is 30 days but your policy states 14 days, you have a source accuracy problem that must be resolved before deployment.

Tone and Voice

Does the response match your brand voice? A legal services firm needs formal, precise language. An e-commerce brand targeting Gen Z might want a conversational, friendly tone. Your system prompt controls this, and the Playground is where you verify it works.

Completeness

Did the agent fully answer the question, or did it provide a partial response? If a customer asks "What payment methods do you accept and is there a minimum order?" the agent should address both parts. Incomplete answers frustrate users and generate follow-up messages.

Source Attribution

Expand the source citations on each response. Verify that the agent is referencing the correct, most relevant source. If the agent answers a pricing question by citing your "About Us" page, your sources may need restructuring or additional Q&A pairs for precision.

Debugging Techniques

When a response does not meet expectations, the Playground provides the tools you need to diagnose and fix the problem.

Check Source Citations

Every response includes expandable citations. Click to reveal which sources the agent consulted. If the wrong source was used, the issue is likely in how your content is chunked or how similar your sources are on that topic. Consider adding a specific Q&A pair for that question.

Identify Knowledge Gaps

If the agent responds with a fallback message like "I don't have information about that," the topic is not covered in your training data. Add a new source, text entry, or Q&A pair that addresses the question, then re-test immediately.

Diagnose Hallucinations

If the agent generates information that is not present in any source, the temperature setting may be too high, or the system prompt may not include strong enough grounding instructions. Add an instruction like "Only answer based on the provided training data. If you do not have the information, say so."

Test Boundary Conditions

Ask the agent questions that are adjacent to your training data but not explicitly covered. This reveals how well the agent generalizes and whether it appropriately declines to answer questions outside its scope.

Testing Scenarios Before Deployment

Run through each of the following scenario categories before making your agent public. Document the results and iterate until every category meets your quality bar.

Adjusting Settings from the Playground

You do not need to leave the Playground to change your agent's configuration. The settings panel lets you modify key parameters and immediately see the effect on responses.

Model Selection

Switch between available models to compare response quality and speed. For example, test with both GPT-4o and GPT-4o Mini to find the right balance of quality and cost for your use case.

Temperature

Adjust the temperature slider to control response variability. A temperature of 0.0 to 0.3 produces highly consistent, deterministic answers ideal for customer support. A temperature of 0.7 to 1.0 produces more varied, creative responses better suited for brainstorming or content generation.

System Prompt

Edit the system prompt directly from the Playground. This is the fastest way to iterate on tone, boundaries, and behavioral rules. Make a change, send a test message, evaluate the result, and repeat.

Conversation History and Session Management

Each Playground session maintains a full conversation history, allowing you to test multi-turn interactions where context builds over time. The agent remembers previous messages in the session, just as it would in a live conversation with a real user.

To start a fresh session, click the Reset button. This clears all conversation history and resets the agent's context window. Use this when you want to test a new scenario without previous messages influencing the response.

Programmatic Testing via API

For teams that want to automate testing or integrate Playground-style interactions into CI/CD pipelines, you can use the Chatsby API to send messages programmatically.

curl -X POST https://api.chatsby.co/v1/agents/YOUR_AGENT_ID/chat \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "What is your return policy?",
    "session_id": "test-session-001"
  }'

The response includes the agent's reply, source citations, and confidence metadata:

{
  "response": "Our return policy allows returns within 30 days of purchase...",
  "sources": [
    {
      "title": "Return Policy",
      "type": "text",
      "relevance_score": 0.94
    }
  ],
  "session_id": "test-session-001",
  "tokens_used": 287
}

Use this endpoint to build automated regression tests that verify your agent's responses remain accurate after source updates or configuration changes.

Best Practices for Iterative Improvement

1. Test after every change. Whether you add a new source, edit your system prompt, or adjust the temperature, always validate in the Playground before deploying.
2. Create a test script. Maintain a list of 20-30 representative questions covering your core use cases, edge cases, and off-topic scenarios. Run through this list after every significant change.
3. Test as the user, not the builder. Ask questions the way your customers would, including typos, incomplete sentences, and colloquial language. Your agent needs to handle real-world input.
4. Use the Reset button between scenarios. Avoid context bleed between unrelated test cases by starting fresh sessions.
5. Document failures. When you find a bad response, note the question, the response, and what you changed to fix it. This becomes your QA log and helps prevent regressions.
6. Involve your team. Have customer-facing team members test the agent. They know the questions customers actually ask and can spot issues you might miss.
7. Compare models. Test the same questions across different model configurations to find the optimal balance of quality, speed, and cost for your specific use case.