File size continuously increase when use h5D.write() to update some data block in original place.

Question

raym 2021-12-13

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1609625-file-size-continuously-increase-when-use-h5d-write-to-update-some-data-block-in-original-place

评论： Walter Roberson 2021-12-13

采纳的回答： Walter Roberson

在 MATLAB Online 中打开

Hi, I am trying to update a data block into a h5 file using h5D.write.

The updated file opens with no problem, but the file size continuously increased about 6MB everytime I update.

The code is below:

% the data block size is [sizeX,sizeY,1], always at the same position.
fid = H5F.open(aPath,'H5F_ACC_RDWR','H5P_DEFAULT');
dset_id  = H5D.open(fid,['/Data']);
block = [sizeX,sizeY,1];h5_block = fliplr(block); %the same size as original file
dims = [sizeX,sizeY,1];
mem_space_id = H5S.create_simple(length(dims),h5_block,[]);
space_id = H5D.get_space(dset_id);
aStart = [0,0,0];h5_start = fliplr(aStart);
H5S.select_hyperslab(space_id,'H5S_SELECT_SET',h5_start,[],[],h5_block);
H5D.write(dset_id,'H5ML_DEFAULT',mem_space_id,space_id,'H5P_DEFAULT',aData); %aData is updated data of same size as original file
H5D.close(dset_id);H5F.close(fid);

Since no size expansion is involved, Is there a way to avoid file size increase?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2021-12-13

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1609625-file-size-continuously-increase-when-use-h5d-write-to-update-some-data-block-in-original-place#answer_853825

Sorry, no.

When you update, HDF5 writes a new block in, and then marks the old block as no longer in use.

Unfortunately it also has no defined garbage collection, so although the next time around the now-unused block would in theory be large enough to store the data to be written, in practice it may not be.

There is no inherent HDF5 operation to reclaim space.

There are some tools that will repack HDF5 files; see https://www.mathworks.com/matlabcentral/answers/395920-how-can-i-delete-a-dataset-completely-from-a-group-in-a-hdf5-file

3 个评论
显示 1更早的评论隐藏 1更早的评论

raym 2021-12-13

but another problem is that the increased file size is much larger than original file size.

It may be caused by the different deflate option.

As this is re-write data block instead of create a data set from begining, Is there a way to indicate the deflate option to updated data?

Walter Roberson 2021-12-13

If the data is being compressed, then remember that a change in value may result in the compressed data being shorter or longer than the previous version.

h5repack appears to support a command line argument to indicate compression type.

https://www.systutorials.com/docs/linux/man/1-h5repack/

For example -f GZIP=9

请先登录，再进行评论。