Read in-memory csv file with open() in python

Question

I have a dataframe. I want to write this dataframe from an excel file to a csv without saving it to disk. Looked like StringIO() was the clear solution. I then wanted to open the file like object from in memory with open() but was getting a type error. How do I get around this type error and read the in memory csv file with open()? Or, actually, is it wrong to assume the open() will work?

TypeError: expected str, bytes or os.PathLike object, not StringIO

The error cites the row below. From the code that's even further below.

f = open(writer_file)

To get a proper example I had to open the file "pandas_example" after creation. I then removed a row, then the code runs into an empty row.

from pandas import util
df = util.testing.makeDataFrame()
df.to_excel('pandas_example.xlsx')
    
df_1 = pd.read_excel('pandas_example.xlsx')

writer_file = io.StringIO()
    
write_to_the_in_mem_file = csv.writer(writer_file, dialect='excel', delimiter=',')
    
write_to_the_in_mem_file.writerow(df_1)
    
f = open(writer_file)

while f.readline() not in (',,,,,,,,,,,,,,,,,,\n', '\n'):
        pass

final_df = pd.read_csv(f, header=None)
    

f.close()

Answer 1

Think of writer_file as what is returned from open(), you don't need to open it again.

For example:

import pandas as pd
from pandas import util
import io

# Create test file
df = util.testing.makeDataFrame()
df.to_excel('pandas_example.xlsx')
    
df_1 = pd.read_excel('pandas_example.xlsx')

writer_file = io.StringIO()
df_1.to_csv(writer_file)
writer_file.seek(0)     # Rewind back to the start    

for line in writer_file:
    print(line.strip())     

The to_csv() call writes the dataframe into your memory file in CSV format.

Answer 2

After

writer_file = io.StringIO()

, writer_file is already a file-like object. It already has a readline method, as well as read, write, seek, etc. See io.TextIOBase, which io.StringIO inherits from.

In other words, open(writer_file) is unnecessary or, rather, it results in a type error (as you already observed).