Create intermediate Works with: Any AI

Large CSV Processing in Chunks

Structured brief Has example

Handle files that do not fit in memory.

The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.

Design a chunked processing approach for a large file. File size, columns and types: [FILE_DETAILS]. Available memory and tools: [MEMORY_TOOLS]. What I need to compute: [GOAL].

First decide whether chunking is the right tool or whether another approach (selecting columns, optimising dtypes, a database, or a columnar format) would be simpler, and explain the reasoning. Then provide code using pandas read_csv with chunksize or usecols and dtype settings, with per-chunk cleaning, running aggregates that can be combined correctly (sums and counts rather than averages of averages), and writing results incrementally. Show how to test the chunked result against a full run on a sample. Add notes on memory profiling, handling headers, and bad rows. List the operations that cannot be done chunk by chunk.

After the code, add a short benchmark plan: time and peak memory for the full approach on a sample versus the chunked approach, using a method I can run on my own machine, and a rule for choosing chunk size by testing three values. Provide a checklist for verifying that no rows were dropped at chunk boundaries and that the combined totals match a direct computation on a small file.

Fill in before use

Optional: type your details and the prompt above updates. Nothing is saved or sent.

Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.

Decision: the file has 40 columns but I need 6; first try usecols and dtype before chunking.
Code sketch:
total = 0; counts = {}
for chunk in pd.read_csv(path, usecols=cols, dtype=types, chunksize=500_000):
    chunk = chunk.dropna(subset=["amount"])
    total += chunk["amount"].sum()
    for k, v in chunk.groupby("region")["amount"].agg(["sum","count"]).iterrows():
        a = counts.get(k, (0, 0)); counts[k] = (a[0]+v["sum"], a[1]+v["count"])
Mean by region = sum / count, never an average of chunk averages.
Test: run on the first 100,000 rows both ways; totals must match to the cent.

• pandas
• structured-brief