'Too many open files in Windows when writing multiple HDF5 files

My question is how to close HDF5 files indefinitely after writing them?

I am trying to save data to HDF5 files - there are around 200 folders and each folder contains some data for each day for this year.

When I retrieve and save data using pandas HDFStore with following code in iPython console, the function stop automatically after a while (no error msg).

import pandas as pd

data = ... # in format as pd.DataFrame
# Method 1
data.to_hdf('D:/file_001/2016-01-01.h5', 'type_1')
# Method 2
with pd.HDFStore('D:/file_001/2016-01-01.h5', 'a') as hf:
    hf['type_1'] = data

When I tried the same script to download data again, it says:

[Errno 24] Too many open files: ...

There are some posts suggesting using ulimit -n 1200 for example in Linux to overcome the problem, but unfortunately I'm using Windows.

Besides, I think I already close files explicitly using with closure, especially in Method 2. How come iPython still count these files as open?

My loop is sth like below:

univ = pd.read_excel(univ_file, univ_tab)
for dt in pd.DatetimeIndex(start=start_date, end=end_date, freq='B'):
    for t in univ:
        data = download_data(t, dt)
        with pd.HDFStore(data_file, 'a') as hf:
            # Use pd.DataFrame([np.nan]) instead of pd.DataFrame() to save space
            hf[typ] = EMPTY_DF if data.shape[0] == 0 else data


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source