Cross-Validation Design for My Data
Choose a validation scheme that matches reality.
The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.
Design a cross-validation scheme. Data description (rows, groups, time, class balance): [DATA_DESCRIPTION]. How the model will be used: [USE_CASE]. Compare k-fold, stratified, grouped, and time-based splits and decide which fits my data and why, with the leakage each avoids. Provide Python code for the chosen scheme using standard tools, with a nested approach if I tune hyperparameters. Explain how to report mean and spread of scores and how many repeats to use. Add a leakage checklist: duplicates across folds, features computed on full data, and preprocessing fitted outside the folds. Include a table of scenarios and the correct split for each. Add a worked example of splitting a dataset of customers with repeated rows for each customer, showing how a plain split leaks and how a grouped split fixes it, with invented numbers. Include a check to compare cross-validated scores with the final test score and an explanation of large gaps. Add a worked comparison on an invented dataset of 10,000 rows from 1,200 customers, in which each customer has several rows. Describe what happens with a plain five-fold split (rows of the same customer appear on both sides, inflating accuracy from, say, 0.78 to 0.86 in the illustration) and with a grouped split, and how to detect the leak by comparing the scores. Label the numbers illustrations. Add a decision flow that picks the split from four facts (groups, time order, class imbalance, small data), a nested cross-validation outline with the roles of inner and outer loops, and a reporting template: mean, standard deviation, number of repeats, the split used and why. Include a final check of a held-out test set used once. Close with a short list of the five numbers to report from the validation (mean, spread, number of folds, number of repeats, size of the final held-out test) so results can be compared fairly later.
Fill in before use
Optional: type your details and the prompt above updates. Nothing is saved or sent.
Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.