Reading binary files consisting of different data types without a for loop.

47 次查看（过去 30 天）

Adam 2014-9-15

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/154920-reading-binary-files-consisting-of-different-data-types-without-a-for-loop

编辑： Mark Thomson 2022-5-2

I have a binary data file that consists of some M data sets. Each set of data is made up of Nbytes of a specific template, e.g. [uint16, uint16, uint16, uint32, double, uint32, int16]. Right now I'm just looping over how many data sets I have and reading the information in each data set according to it's type.

for j = 1:Mdatasets
     this(j) = fread(fid,1,'uint32');
     foo(j) = fread(fid,1,'uint16');
     foofoo(j) = fread(fid,1,'double');
     % and so on...
end

Is there a faster way to do this? It can't take a very long time to read some of my larger files (~500MB). I was thinking that if you could give fread() a data type template to repeat over and over like it can do with a single data type, that would be ideal. Not sure if there is a way to do this, or if someone has a way around it, but for loops take so long.

Best Regards,

Adam

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Image Analyst 2014-9-15

The for loop is definitely NOT the problem You can do tens of millions of iterations in less than a second and I'm sure you don't have that many files. The time is being taken up by the disk I/O rather than the for loop.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（2 个）

Guillaume 2014-9-15

2
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/154920-reading-binary-files-consisting-of-different-data-types-without-a-for-loop#answer_151787

编辑：Guillaume 2014-9-17

在 MATLAB Online 中打开

Use the skip argument of fread to read all the elements of the same type at once. From my reading of fread doc, skip is not very straightforward to calculate. This may work:

fieldsizes = [4 2 8 ...]; %uint32 uint16 double ...
skip = @(n) sum([fieldsize(1:n+1) fieldsize(n+1:end)]); %sum up of field sizes except field n
offset = @(n) sum(fieldsizes(1:n)); %offset to element n+1
this = fread(fid, Mdatasets, 'uint32', skip(1));
fseek(fid, offset(1), -1);
foo = fread(fid, Mdatasets, 'uint16', skip(2));
fseek(fid, offset(2), -1);
foofoo = fread(fid, Mdatasets, 'double', skip(3));
fseek(fid, offset(3), -1);
...

edit: added fseek as per Michael comment.

6 个评论
显示 4更早的评论隐藏 4更早的评论

Guillaume 2014-9-17

Thanks Michael for the reminder about the file pointer, I'd completely missed that. It's actually fseek that's needed since you need to go back to the right element.

Reading the file my or Iain's way is bound to be faster than reading it one element at a time.

A third option, probably the fastest is to write a mex file.

Mark Thomson 2022-5-2

编辑：Mark Thomson 2022-5-2

Many thanks Guillaume for pointing out the Skip functionality (I should have read the fread help in full first)... I had a similar issue, only with Fortran UNFORMATTED files where this helped skip the record entries.

It seems that the overheads in a for loop are indeed due to the repetitive file access, not being streamlined by the functionality of the compiled built-in fread with Skip.

请先登录，再进行评论。

Image Analyst 2014-9-15

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/154920-reading-binary-files-consisting-of-different-data-types-without-a-for-loop#answer_151786

在 MATLAB Online 中打开

You're reading 1 byte at a time - no wonder it takes so long. Read in a whole image at a time:

thisImage = fread(fid, [rows, columns], '*uint16');

5 个评论
显示 3更早的评论隐藏 3更早的评论

Image Analyst 2014-9-16

If you have patterns of data in one file, like a bunch of small variables (header info) and then maybe a big image, then you could make a subroutine to do the "common" part. It could take in a page or slice number and use fseek to go to the starting point for that page/slice. I've written readers for custom image formats, like CT data, and I could give you examples if you want. I read in the header and image data.

Adam 2014-9-16