Batch Job and ETL Performance Review
Speed up scheduled data processing.
The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.
ROLE: You are a data engineer who tunes batch pipelines. CONTEXT: Job: [JOB_NAME]. Data volume: [DATA_VOLUME]. Current duration: [CURRENT_DURATION]. Time window: [TIME_WINDOW]. Step timings: [PASTE_STEP_TIMINGS]. Code or SQL excerpts: [PASTE_CODE_OR_SQL]. STEPS: 1. Find the dominant step. 2. Evaluate chunking, parallelism, bulk operations versus row-by-row, index usage, temporary tables, staging, incremental processing instead of full reloads, network transfer, and memory pressure. 3. Propose an optimized design and estimate the new duration with assumptions stated. 4. Describe how to make the job restartable and idempotent. 5. Provide a monitoring dashboard specification and a safe cutover plan. OUTPUT FORMAT: Bottleneck analysis, redesign, duration estimate, and cutover plan. QUALITY BAR: Correctness of results is non-negotiable; define how to prove outputs match the old job. Define output equivalence tests: row counts, checksums on key columns, and a sample comparison on edge cases such as nulls and time zones. Recommend running old and new jobs side by side for at least [PARALLEL_RUN_DAYS] days. FOLLOW-UP: Ask me what downstream reports depend on this job.
Fill in before use
Optional: type your details and the prompt above updates. Nothing is saved or sent.
Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.