Create intermediate Works with: Any AI

A/B Test Plan and Sample Size

Structured brief Has example

Plan a test before collecting data.

The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.

Plan an A/B test. Metric and baseline: [METRIC_AND_BASELINE]. Smallest improvement worth acting on: [MINIMUM_EFFECT]. Traffic available per week: [TRAFFIC]. Risks and constraints: [RISKS_AND_CONSTRAINTS].

Produce:
1. A hypothesis statement and the decision rule decided in advance (ship if ..., stop if ..., extend if ...).
2. Sample size per group for a two-proportion test using n = 2 x (z_alpha + z_beta) squared x p(1-p) / delta squared, with z values for 5 percent significance and 80 percent power (1.96 and 0.84), worked with my numbers, and the duration in full weeks. Say this is approximate and suggest verifying with a power calculator.
3. Randomization unit and how to avoid contamination (user level, not session level).
4. Guardrail metrics and stop conditions for harm.
5. A pre-registration table: Primary metric | Secondary metrics | Segments allowed | Analysis method | Stopping rule.
6. Pitfalls: peeking, novelty effects, sample ratio mismatch (check with a chi-square test), multiple variants, seasonality.
7. A one-page results template with the effect size, interval, and decision.
If the traffic cannot support the test, say so and propose alternatives.

Fill in before use

Optional: type your details and the prompt above updates. Nothing is saved or sent.

Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.

Baseline conversion 5.0%; minimum detectable effect 0.5 points (to 5.5%); alpha 0.05; power 80%.
Sample per group (two proportions) ≈ 16 x p(1-p)/d^2 = 16 x 0.0475 / 0.000025 ≈ 30,400 [rule of thumb; confirm with a calculator].
Traffic 4,000 visitors per week per group → about 8 weeks. Too long? Options: accept a larger MDE of 0.8 points (about 12,000 per group, 3 weeks), or test a bigger change.
Metrics: primary conversion; guardrail: refund rate; secondary: revenue per visitor.
Rules: no peeking before the planned end; run full weeks.
Decision table: win → ship; loss → revert; flat → keep simpler version.

• ab-planning
• structured-brief