Delay reading/writing files after fwrite returns

Question

Adam Ranson 2017-12-1

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/370527-delay-reading-writing-files-after-fwrite-returns

编辑： Greg 2017-12-2

image.png

Hi I'm trying to understand why fread might be taking longer when followed by fwrite when reading and writing files of ~5GB.

If I generate 4 files of ~5GB like this:

paths{1} = 'e:\temp\f1.bin';
paths{2} = 'e:\temp\f2.bin';
paths{3} = 'e:\temp\f3.bin';
paths{4} = 'e:\temp\f4.bin';
arr = ones(1,30e8,'int16');
for i = 1:4
    i
    fid = fopen(paths{i},'w');
    fwrite(fid, arr,'int16');
    fclose(fid);
end

And then try to read them and write back to them like this:

paths{1} = 'e:\temp\f1.bin';
paths{2} = 'e:\temp\f2.bin';
paths{3} = 'e:\temp\f3.bin';
paths{4} = 'e:\temp\f4.bin';
readSpeed = [];
writeSpeed = [];
figure;
for iFile = 1:4
    disp(['Reading file ',num2str(iFile)]);
    tic;
    fid = fopen(paths{iFile});
    arr = fread(fid, inf, ['*','int16']);
    fclose(fid);
    readSpeed(end+1) = toc;
    disp(['Writing file ',num2str(iFile)]);
    tic;
    fid = fopen(paths{iFile},'w');
    fwrite(fid, arr,'int16');
    fclose(fid);
    writeSpeed(end+1) = toc;
    hold off;
    plot(readSpeed,'*b');
    hold on;
    plot(writeSpeed,'*r');
    ylim([0 max([readSpeed,writeSpeed])*1.2]);
    drawnow;
end

I get the behaviour in the attached plot - they all write at the same speed but the first file read takes about half as long as the rest.

Any suggestions much appreciated

Adam

2 个评论
显示无隐藏无

Greg 2017-12-2

编辑：Greg 2017-12-2

I don't understand the question title: fread slower when followed by fwrite All of the freads in your example are followed by an fwrite, and only the first one is faster.

To point (why is it so drastically different): there is a LOT going on. As per Isakson mentioned, you have the entire inner workings of the OS to worry about. Not to mention, you have MATLAB's JIT and hardware capabilities.

I'm most concerned that the fread takes longer than the fwrite at all. I don't think I've ever seen a hard drive with a higher write speed than read.

Some possibilities: hard drive cache, memory cache, pagefile (virtual RAM). The times reported by tic and toc are when MATLAB thinks the operation is complete. This is quite often long before it actually completes. MATLAB tells the OS "have some stuff to write to a disk" then the OS usually stuffs that in a buffer or cache to wait for the disk to fully write it. As soon as the entire chunk is in cache, MATLAB gets the "done" and toc tells you a time.

I suspect your 2:4 fread times are actually coming from the preceding fwrite statements. The fwrite hits the cache, MATLAB thinks it's done and calls toc. The fread asks for data, the OS says "Oh huh uh! I'm still writing, you gotta wait." So the fread tic just keeps climbing while the cache is flushed from the previous fwrite.

Adam Ranson 2017-12-2

Thanks for the responses. I was confused at why writing seemed to be taking less time than reading. I used this title because when I did writing alone it always took the same time (~45 secs for this file). I've run the code with just reading, or just writing and your explanation is correct - the fwrite function returns once the data is sent off to be written, but there's a delay if you then try to read or write a new file afterwards while the OS etc is actually continuing to do the writing. These are the timings if you just do reads or just do writes:

I'll try to change the title to make it more informative. Thanks again

Adam

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Greg 2017-12-2

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/370527-delay-reading-writing-files-after-fwrite-returns#answer_294226

编辑：Greg 2017-12-2

See the comment chain for gory details, but here's the bottom line:

I think the best option for testing the actual read/write time would be to include one more fread & fwrite pair of 1 byte inside each of your tic/toc timers. Reading and writing a byte won't impact your measurement, but it will force the OS to flush the 5GB fread or fwrite and should give you a realistic measurement.

Always happy to help.