H5 file reading very slow with Java

Question

I have a Java program using the H5 libraries that tries to read a dataset in a H5 file with the following properties:

The file's size is 769M.

The code that reads the dataset is the following (very simple):

// Open file using the default properties.
fileId = H5.H5Fopen(filepath, HDF5Constants.H5F_ACC_RDONLY, HDF5Constants.H5P_DEFAULT);
// Open dataset using the default properties.
if (fileId >= 0) {
   datasetId = H5.H5Dopen(fileId, "/data/0_u0/20050103", HDF5Constants.H5P_DEFAULT);
}

if (datasetId >= 0) {
   dataSpaceId = H5.H5Dget_space(datasetId);
}

// Get the dimensions of the dataset
int ndims = -1;
if (dataSpaceId >= 0)
   ndims = H5.H5Sget_simple_extent_ndims(dataSpaceId);

if (ndims > 0) {
    long[] dims = new long[ndims];
    H5.H5Sget_simple_extent_dims(dataSpaceId, dims, null);
    H5.H5Sclose(dataSpaceId);

    int dimX = (int)dims[0];
    int dimY = (int)dims[1];

    Double[][] dsetData = new Double[dimX][dimY];
    H5.H5Dread(datasetId, HDF5Constants.H5T_NATIVE_DOUBLE,
               HDF5Constants.H5S_ALL, HDF5Constants.H5S_ALL,
               HDF5Constants.H5P_DEFAULT, dsetData);
}

And it takes forever (more than 15 minutes, I stopped after that). What I don't understand is that I also have kind of the same code in Python, and it takes a few seconds.

When I debug the Java program and stop in the middle execution, it's in the byteToDouble() function of the H5 lib. It's a lot of double, but should not take that much time right?

Thanks for your help!

Answer 1

I think the issue is that your reading the data into 2D array Double[][]. When you do this the HDF5 implementation is very slow (think the issue is probably in HDFArray.arrayify). Try reading the data into a 1D double[].

Also you are using boxed Double it would probably be better to use primative double.