'Multiprocessing: writing into a hdf5 file
I am running a parallelized code in Python and I am trying to save some values within each iteration. My code could be simplified/summarized as follows:
# Import necessary libraries
def func(a,b):
# Generate some data and save it into "vector".
# Create a Hdf5 file and save data in vector.
with h5py.File('/some_file.hdf5', 'w') as f:
f.create_dataset('data_set', data=vector)
# Some code
# Parallelize func
if __name__ == '__main__':
with mp.Pool(2) as p:
[p.apply_async(func, args=(elem, b)) for elem in big_array]
I am saving the files while parallelizing to save memory, since I will be working with big amounts of data.
However, every time I run the script, no hdf5 file is generated and the data is not saved.
I am pretty new to Parallelization with Python and I do not understand what the problem is.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
