'H5PY storage device disconnections management

I have a Python program that collects data from electronic devices at a defined frequency. Each time I get new data, I have to store them into an HDF5 file that must be saved on a USB drive.

I want to manage this drive disconnections, but every time I unplug the USB drive while h5py has a handle on it, it crashes without letting me catch the exception.

Below is an example of the behaviour:

import random
import h5py

try:
    h5f = h5py.File(r"E:\data.h5", "a")
    try:
        dset = h5f["dataset"]
    except KeyError:  # dset does not exist yet => create it
        h5f.create_dataset("dataset", (0,), maxshape=(None,), chunks=(1024,))
        dset = h5f["dataset"]
        
    input("waiting...")

    dset.resize((dset.shape[0] + 1,))
    dset[-1] = random.random()
except Exception:
    print("exception raised")
else:
    h5f.close()

If I run this script and unplug the E: drive when waiting... is displayed, then I got the following output:

$ python tmp.py
waiting...
exception raised
Traceback (most recent call last):
  File "h5py\_objects.pyx", line 193, in h5py._objects.ObjectID.__dealloc__
RuntimeError: Can't decrement id ref count (file write failed: time = Wed May 11 16:09:20 2022
, filename = 'E:\data.h5', file descriptor = 3, errno = 22, error message = 'Invalid argument', buf = 000002573736C038, total write size = 272, bytes this sub-write = 272, bytes actually written = 18446744073709551615, offset = 800)
Exception ignored in: 'h5py._objects.ObjectID.__dealloc__'
Traceback (most recent call last):
  File "h5py\_objects.pyx", line 193, in h5py._objects.ObjectID.__dealloc__
RuntimeError: Can't decrement id ref count (file write failed: time = Wed May 11 16:09:20 2022
, filename = 'E:\data.h5', file descriptor = 3, errno = 22, error message = 'Invalid argument', buf = 000002573736C038, total write size = 272, bytes this sub-write = 272, bytes actually written = 18446744073709551615, offset = 800)

When this happens, I have not found any way to catch the h5py exceptions in order to keep my program alive and continue processing.

Is there anything I can do?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source