Hi Guillaume,
I see that you are experimenting with using hdf5 files for your project. The answers to both of your queries are as follows:
Firstly, using the low level h5 functions can lead to some improvement in the performance. But the complexity of the code will also increase proportionately. With low level functions you can have a fine-grained control over the data access, custom tuning of data chunk sizes and an easier implementation of parallel I/O operations. But all of this requires a deeper understanding of the functions and will considerably complicate your code.
Secondly, it would be better to use a single hdf5 file instead of multiple files because of the following reasons:
- File system overhead: Every file access has a certain overhead and multiple files will cause the code to be significantly slower.
- Better I/O optimization: A large hdf5 file can provide better optimizations like reading and writing larger, contiguous blocks in the file.
- File management: It is easier to keep track of one file instead of thousands of files.
Also, you can have parallel access to a single hdf5 file using the single writer multiple readers (SMWR) model in MATLAB. Using this functionality, you can read in parallel, but you can't write in parallel. More information about this functionality is provided in the following documentation:
I hope it helps.
