Checking file or data integrity

Question

Sasha Egan 2018-7-19

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/411175-checking-file-or-data-integrity

回答： Giresse 2024-4-23，2:42

G'day mates, I am collecting data from a multi-channel DAQ, therefore, the datasets tend to be quite large. I am splitting each channel out from the lumped data into its own respective vector and saving an interim copy as a .mat binary. Is there a way to verify that the data saved in the .mat binary and the vector in memory are identical? i.e. ( in pseudo-code )

A = [ channel 1 ]
A_check = checksum (a)
save A as A.mat
A.mat_check = checksum(A.mat)
assert {
  A_check equals A.mat_check
} throw exception e1 CheckSumsDoNotMatch
if (A_check != A.mat_check)
...try again...

Cheers.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Jan 2018-7-20

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/411175-checking-file-or-data-integrity#answer_329610

编辑：Jan 2018-7-20

在 MATLAB Online 中打开

Maybe with FEX: DataHash or FEX: GetMD5:

A = rand(1, 1e6);
A_check = DataHash(A);   % Or: GetMD5(a, 'Binary')
save('A.mat', A);
FileData = load('A.mat');
A_reloaded_check = DataHash(FileData.A);
if ~isequal(A_check, A_reloaded_check)
   error('Saving failed!');
end

You should definitely stop with an error. If the saving fails, there is a severe problem, which cannot be handled reliably by repeating. Either the disk is damaged or the CPU melts down.

[EDITED] Guillaume is right: Calculating the checksums has no advantage. I suggested it only, because there might be a need anywhere, which was not mentioned in the question. An efficient version of my code would be:

A = rand(1, 1e6);
save('A.mat', A);
FileData = load('A.mat');
if ~isequal(A, FileData.A)
   error('Saving failed!');
end

2 个评论
显示无隐藏无

Guillaume 2018-7-20

编辑：Guillaume 2018-7-20

在 MATLAB Online 中打开

I don't really see the point in computing the hashes. It's just going to take as long as simply comparing the matrices themselves. After loading the mat file, you could just do:

if ~isequal(A, FileData.A)
  error(...)

What's puzzling me is why the question in the first place? If you suspect that for some reason the data does not survive the round trip through a mat file, then you should have procedures in place to detect hard disk corruption and recover from such. That should happen at the OS or hardware level, not within matlab.

matlab itself, will not corrupt a mat file.

Sasha Egan 2018-7-20

To allay your puzzle. Because I was writing directly to a network drive. Stuff happens in between sometimes. I was looking for an easy way of checking that the write was successful without resorting to writing the data locally...thank you for the suggestions. :-)

请先登录，再进行评论。

Answer 2

Giresse 2024-4-23，2:41

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/411175-checking-file-or-data-integrity#answer_1446066

在 MATLAB Online 中打开

A = rand(1, 1e6);
A_check = DataHash(A);   % Or: GetMD5(a, 'Binary')
save('A.mat', A);
FileData = load('A.mat');
A_reloaded_check = DataHash(FileData.A);
if ~isequal(A_check, A_reloaded_check)
   error('Saving failed!');
end

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 3

Giresse 2024-4-23，2:42

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/411175-checking-file-or-data-integrity#answer_1446071

在 MATLAB Online 中打开

A = rand(1, 1e6);
A_check = DataHash(A);   % Or: GetMD5(a, 'Binary')
Unrecognized function or variable 'DataHash'.
save('A.mat', A);
FileData = load('A.mat');
A_reloaded_check = DataHash(FileData.A);
if ~isequal(A_check, A_reloaded_check)
   error('Saving failed!');
end