Create intermediate Works with: Any AI

Prompt A/B Comparison Plan

Structured brief Has example

Compare two prompt versions fairly.

The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.

Plan a fair comparison between two prompts. Prompt A: [PROMPT_A]. Prompt B: [PROMPT_B]. The task and inputs: [TASK_AND_INPUTS]. What better means for us (accuracy, tone, brevity, safety, cost): [DEFINITION_OF_BETTER].

Return:
1. Hypothesis: what I expect B to do better and why.
2. Test design: at least 20 varied inputs; run each prompt on each input the same number of times with identical settings; randomize the order; hide which prompt produced which output when grading.
3. A scoring sheet: criteria with 1 to 5 anchors (describe a 1, 3, and 5 for each), plus pass or fail checks for must-haves.
4. Analysis: average scores per criterion, win-loss counts per input, and a look at the worst outputs from each, not just averages.
5. Decision rule written in advance (for example B must beat A on accuracy without a drop in safety checks, with a difference clearly larger than the variation between repeated runs).
6. Cost comparison: length of the prompt and outputs, and time per run.
7. Logging template: prompt version, date, tool, settings, results, notes.
8. Pitfalls: testing on only the examples you tuned for, grading when you know which is which, and changing two things at once.

State clearly that small tests show direction, not proof. Provide a short report template.

Fill in before use

Optional: type your details and the prompt above updates. Nothing is saved or sent.

Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.

Hypothesis: B (adds an example and a length limit) improves format compliance.
Design: 20 inputs, each run 3 times per prompt, same settings, order randomized, grader blind.
Scoring sheet: accuracy 1-5, format compliance pass/fail, tone 1-5.
Decision rule (written before testing): B must improve accuracy by at least 0.5 points with no drop in safety checks, and the difference must exceed the variation between repeated runs of A.
Results: A mean 3.6, B mean 4.1 (made-up). Worst outputs reviewed. Small tests show direction, not proof.

• ab-testing
• structured-brief