'Too many open files in Windows when writing multiple HDF5 files
My question is how to close HDF5 files indefinitely after writing them?
I am trying to save data to HDF5 files - there are around 200 folders and each folder contains some data for each day for this year.
When I retrieve and save data using pandas HDFStore with following code in iPython console, the function stop automatically after a while (no error msg).
import pandas as pd
data = ... # in format as pd.DataFrame
# Method 1
data.to_hdf('D:/file_001/2016-01-01.h5', 'type_1')
# Method 2
with pd.HDFStore('D:/file_001/2016-01-01.h5', 'a') as hf:
hf['type_1'] = data
When I tried the same script to download data again, it says:
[Errno 24] Too many open files: ...
There are some posts suggesting using ulimit -n 1200 for example in Linux to overcome the problem, but unfortunately I'm using Windows.
Besides, I think I already close files explicitly using with closure, especially in Method 2. How come iPython still count these files as open?
My loop is sth like below:
univ = pd.read_excel(univ_file, univ_tab)
for dt in pd.DatetimeIndex(start=start_date, end=end_date, freq='B'):
for t in univ:
data = download_data(t, dt)
with pd.HDFStore(data_file, 'a') as hf:
# Use pd.DataFrame([np.nan]) instead of pd.DataFrame() to save space
hf[typ] = EMPTY_DF if data.shape[0] == 0 else data
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
