How to access and manipulate each chunk in python?

Question

I created the following chunks

import pandas as pd
import pyodbc
df = pd.read_sql_query(address,cnxn,chunksize=10000000)

Now, I want to access each chunk and do some analysis on them
The following method works on small datasets, but again the list growing bigger and bigger and it gives memory error on large datasets:

for chunk in df:
    dfl.append(chunk)
dfl[0] # here I can access the first dataframe in chunks

Is there any other way to access and retrive data when needed? For example, I want to do the following manipulation on spesific chunk:
chunk1['adress'] = chunk1['adress'].str.lower()

How would I do that?

Answer 1

Use:

df = pd.read_sql_query(address,cnxn,chunksize=10000000)
for chunk in df:
    #your code for analyze
    chunk['adress'] = chunk['adress'].str.lower()
    ...
    print (chunk)