Converting from TextFileReader object to pandas DataFrame

Question

I have this code:

f = pd.read_csv(data,delimiter=",",chunksize=1000000)
print(f)
f.head()

Which uses pandas to read the csv file with name from the variable data.

I cannot use the head function as it is a TextFileReader object (the output of print(f) is "pandas.io.parsers.TextFileReader object at 0x78a9180da6d8”)

The error I receive is: AttributeError: 'TextFileReader' object has no attribute 'head'

How do I convert from this object to a pandas dataframe?

Answer 1

This is to do with the Chunk argument. The TextFileReader object contains the chunks and so you must use:

for chunk in f:
    print(chunk) #or whatever other command

As suggested by User Jon Clements:

https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#iterating-through-files-chunk-by-chunk

Has full documentation on this area.