Evaluation
RAG evaluation questions for support, docs, and internal bots
Question templates small teams can use to test whether important RAG answers still retrieve the right context.
Good RAG evaluation questions are real user questions tied to important source content, expected evidence, and a pass or fail threshold for retrieved context.
What makes a good RAG test question?
A good RAG test question should be specific enough to retrieve the right source, important enough to matter if it fails, and stable enough to compare over time.
- Support: What is the current refund policy for annual plans?
- Docs: How do I rotate an API key in the latest SDK?
- Internal knowledge: What is the current onboarding checklist for a new contractor?
- Sales: What is included in the Solo plan versus Builder Pro?
- Codebase: Which endpoint approves a repair from an external agent?
- Product support: Which products are discontinued or unavailable in this region?
How should teams organize retrieval tests?
Group tests by user journey, risk level, and source owner. A small workspace can start with five high-risk questions. Add more when a failed answer would cost support time, revenue, or user trust.
What should happen when a test fails?
A failed test should create a repair item with the failing question, expected source, retrieved source, score, and recommended fix. It should also send an event to the chosen action channel.
Common questions
How many RAG evaluation questions should a small team start with?
Start with five to ten questions that cover high-risk product, support, docs, onboarding, or pricing answers.
Should RAG evaluation be manual or automated?
The scan should be automated, but teams should keep humans in the loop for repair approval and important judgment calls.
What is the difference between retrieval tests and chat transcripts?
Retrieval tests are stable checks that run repeatedly. Chat transcripts are useful raw material, but they need to be turned into expected questions and evidence before they become tests.