Datastore for reading large binary files versus incremental fread

5 次查看（过去 30 天）
gujax
gujax 2021-8-22
评论： gujax 2021-8-23
I am trying to understand why should I be using Datastore to read 100Gb binary file (other than exceeds memory capacity of RAM)?
For one filedatastore reads the full file before splitting up so that does not help me at all (Matlab 2018version). I cannot even read the full data set because it is huge. So why was it even introduced in the first place? I thought the whole point was to resolve the memory issues.
I could create a custom datastore, but then how is that different from using fseek and fread in a for-loop?
Any examples for reading binary large data in help or documentation? If I do create a customized data store as given in the documentation - do I have to run it in a for-loop? Its not clear to me from reading it here
https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.filedatastore.html
For example this code runs too slow for an 8Gb file
1. Do I have to open and close the file everytime? Can I just leave it open?
Why do I need fds object when I can simply do the same with fread and fseek in the for-loop below?
sz = 128*1e6*1.024;
fds = fileDatastore(location,'ReadFcn',@(filename) MyReadFcn(filename,offset,sz));
Dat =[];%should pre-allocate this
for kf = 1:10
offset = sz;
while hasdata(ds)
data = read(fds);
Dat=cat(1,data,Dat);
end
reset(ds)
end
function data = MyReadFcn(filename,offset,sz)
fid=fopen(filename,'rb');
% seek to the offset
fseek(fid,offset,'bof');
% read amount of data
[data] = fread(fid,sz,'*uint8');
fclose(fid);
end
Thank you

请先登录，再回答此问题。

采纳的回答

Walter Roberson
Walter Roberson 2021-8-22
If you use https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.filedatastore.html with ReadMode 'byte' or 'partialfile' then the datastore facility will ask to read only a portion of the file at a time, and will automatically keep track of where you are in the file.
Can you just leave the file open? Yes.
If your datastore has more than one file, then you can
persistent fid previous_filename
if isempty(fid) || ~strcmp(filename, previous_filename)
fclose(fid);
fid = fopen(filename);
previous_filename = filename;
end
You could also detect end-of-file, in which case you would
fclose(fid);
fid = [];

更多回答（1 个）

dpb
dpb 2021-8-22
Basically, the datastore objects are higher-level abstractions for more general file collections. If you can read a portion of the file directly as in your example, you can indeed cut out the middle-man and handle all the details yourself.
It's an attempt at a more user-friendly route -- but, "there is no free lunch!" so there is an overhead to be paid.
I've not had much occasion to play with; but it was my impression that the need to read the full file first was no limitation--if that were the case, it would indeed seem to negate the whole point.
An alternative you might consider for such a file would be memmapfile

类别

在 Help Center 和 File Exchange 中查找有关 Text Files 的更多信息

标签

产品


版本

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by