Evaluation Rubric for Agent Answers

Write a rubric that scores helpfulness, accuracy, safety and honesty.

Added October 15, 2026. Written with AI assistance and checked for clarity and placeholders; not reviewed by a lawyer, accountant, clinician or other professional. How these prompts are made

The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.

Write an evaluation rubric for an agent that answers [DOMAIN_QUESTIONS]. Quality dimensions that matter most: [PRIORITY_DIMENSIONS]. Reviewers: [REVIEWER_TYPES]. Past mistakes: [PAST_MISTAKES].

Define five dimensions: correctness against sources, completeness, clarity, safety and policy compliance, and honesty about uncertainty and actions taken. For each give a 1 to 5 scale with descriptors, an anchor example for 1, 3 and 5, and the evidence a reviewer should cite. Add automatic checks (format valid, citations present, no banned phrases) and a decision rule: any safety fail is a fail regardless of other scores. Provide a scoring sheet and a procedure for two reviewers to calibrate on ten samples and resolve disagreements. Create five invented answers of varying quality to practise on, with the expected scores.

Include guidance on avoiding bias toward long answers and a way to track average scores over versions.

Add a worked scoring example: an answer that is correct but cites no source, is clear, safe, and honest about one unknown scores correctness 4, completeness 3, clarity 5, safety 5, honesty 4, and explain why the missing citation lowers two scores. State how to treat a perfect score on four dimensions and a failure on safety.

Fill in before use

Optional: type your details and the prompt above updates. Nothing is saved or sent.

Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.