Open Text Response Quality and Coding Reliability Check
Audit qualitative coding.
The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.
Audit how consistently our team coded open-ended survey responses. Research question: [RESEARCH_QUESTION]. Our codebook (code names and definitions): [PASTE_CODEBOOK]. A sample of responses with the codes two coders assigned: [PASTE_CODED_SAMPLE]. Learn from this example first: Response: "The app is fine but I can never find my old invoices." Coder A: Usability. Coder B: Missing feature. Analysis: both are defensible; the codebook does not say whether findability counts as usability or as a feature gap. Fix: add a rule and an example to each definition. Now do the same with my sample. Return: 1. A table of agreement: Response ID | Coder A | Coder B | Agree? | Probable reason for disagreement. 2. Percent agreement, and an explanation of how to compute Cohen's kappa step by step with my counts, if I supply them. State that percent agreement alone overstates reliability. 3. The three codes with the most disagreement, why, and the rewritten definition with an include/exclude rule and two examples each. 4. Responses that fit more than one code, and a rule for multiple coding. 5. A calibration routine: coders independently code 20 new responses, compare, discuss, and update the codebook. Do not recode the responses yourself except as examples.
Fill in before use
Optional: type your details and the prompt above updates. Nothing is saved or sent.
Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.