Flaky Test Triage Protocol

Find, classify and fix intermittent CI failures.

Added October 15, 2026. Written with AI assistance and checked for clarity and placeholders; not reviewed by a lawyer, accountant, clinician or other professional. How these prompts are made

The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.

Create a triage protocol for flaky tests in my pipeline. Test framework: [TEST_FRAMEWORK]. Recent failure log excerpts: [LOG_EXCERPTS]. How often builds fail without code changes: [FLAKE_RATE]. Parallelism and shared resources used by tests: [SHARED_RESOURCES].

Work in order. 1. Classify each failure in the logs as timing, ordering, shared state, external dependency, resource limits, or unknown, and quote the evidence line for each. 2. Give me a command or CI step to rerun the suspect test 50 times in isolation and in the full suite, and how to read the difference. 3. For each class list the three most likely root causes and a fix pattern with a short code sketch in my language. 4. Propose a quarantine policy: how long a flaky test may be quarantined, who owns it, and how to track it. 5. Provide a dashboard definition of three metrics (flake rate, mean time to fix, quarantined count). Avoid fixes that only add retries unless you label them as a temporary mitigation.

After step 5, add a worked example using one invented failure: a test that passes alone but fails in the full suite about one run in twelve because two tests write to the same temporary directory. Show the evidence line you would quote, the classification, the isolation command, the likely root causes ranked, the fix pattern, and what the rerun results should look like once fixed (for example, zero failures in 200 full-suite runs). Add a short decision rule for when to delete a flaky test instead of fixing it (it protects no unique behaviour, or the cost of the fix exceeds the cost of the risk), who may approve a deletion, and where the decision is recorded. End with a one-page weekly report format that lists new flaky tests, tests fixed, tests still quarantined with their age, and the single most disruptive test of the week.

Fill in before use

Optional: type your details and the prompt above updates. Nothing is saved or sent.

Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.

Log evidence:
test_checkout_total: passes 46/50 alone, 31/50 in the full suite.
Classification: ordering/shared state. Evidence line: "AssertionError: cart not empty" after test_add_item runs first.
Rerun commands: repeat the test 50 times alone, then 50 times with the whole file shuffled.
Fix pattern: reset the cart fixture per test, remove module-level state:
@pytest.fixture(autouse=True) def clean_cart(): cart.clear()
Quarantine policy: up to 14 days, owner named, ticket required.
Metrics: flake rate (3.1% this week), mean time to fix (6 days), quarantined count (4).

• CI