Why fwrite is ~320x slower in the second interation and onwards when writing interleaved data?

Question

Kouichi C. Nakamura 2019-7-22

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/472847-why-fwrite-is-320x-slower-in-the-second-interation-and-onwards-when-writing-interleaved-data

评论： Kouichi C. Nakamura 2019-7-23

I'd like to write a binary file, which should contain four vectors of integers in the same length in an interleaved fashion.

For example, let si,j denote the sample from the i th channel at time step j.

I can prepare a long vector just as above and then pass it to fwrite and that is easier.

But that design is not great, because when individual vectors are really long, and when I have to accomodate a lot of vectors rather than just four, it will enconter a memory issue.

So, a clever way to to this is to write one vector input at a time without keeping all of them in the memory.

The code like below works, but I found that the performace of fwrite is much worse in the second iteration onwards compared to the first iteration.

While t3 in the first loop was less than a second (0.281 sec), t3 took roughly 1 min 30 sec for the second to fourth iterations (~320 times slower). I wonder why this is so slow, and if there is a work around for this.

fid = fopen(newfile,'w');
for i = 1:4
    filepath = fullfile(datadir,filenames{i});
    
    data = load_data(filepath); % int16 vector
        
    status = fseek(fid,(i-1)*2,'bof');
    if status == -1
        error('fseek failed')
    end
    
    t1 = datetime;
    fwrite(fid, data, 'int16', 2*(n-1));
    t2 = datetime;
    t3 = duration(t2-t1) % why this is slower in second iteration and onwards?   
end
fclose(fid);

t3 = duration

00:00:00.2810

t3 = duration

00:01:37.2600

t3 = duration

00:01:29.9700

t3 = duration

00:01:29.6490

90 (sec) / 0.281 (sec) == ~320 (times)

A simplified code for testing

The longer the vectors are the slower the second and third iterations become, suggesting the slowness has something to do with data before reaching the end of file rather than adding extra bytes at the end of the file.

len = 1000;
i1 = ones(len,1,'int16');
i2 = ones(len,1,'int16').*2;
i3 = ones(len,1,'int16').*3;
fid  = fopen('temp.bin','w')
t1= datetime;
fwrite(fid,i1,'int16',2*2);
t2 = datetime;
t3(1,1) = duration(t2-t1);
t1= datetime;
fseek(fid,2, 'bof')
fwrite(fid,i2,'int16',2*2);
t2 = datetime;
t3(2,1) = duration(t2-t1);
t1= datetime;
fseek(fid,2*2, 'bof')
fwrite(fid,i3,'int16',2*2);
t2 = datetime;
t3(3,1) = duration(t2-t1);
fclose(fid);
t3.Format = 'dd:hh:mm:ss.SSSS'

len = 1000

t3 = 3×1 duration array

00:00:00.0070

00:00:00.0140

00:00:00.0190

2~3 times slower

len = 10000

t3 = 3×1 duration array

00:00:00.0060

00:00:00.0850

00:00:00.1050

14~18 times slower

len = 100000

t3 = 3×1 duration array

00:00:00.0280

00:00:00.9820

00:00:00.9080

35 times slower

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Guillaume 2019-7-22

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/472847-why-fwrite-is-320x-slower-in-the-second-interation-and-onwards-when-writing-interleaved-data#answer_384221

I would suspect the reason for the much slower writes in iteration 2 and onward is that for the first iteration, since the file is new, you're just writing (your data, and 0s in between). From iteration 2, matlab must first read the relevant part of the file, insert the new content and rewrite that.

If you can't fit the whole interlaced data in memory in one go, then you should read the input data in chunk, interlace it in memory, then write it as one continuous block. Repeat for the next chunk of data.

7 个评论
显示 5更早的评论隐藏 5更早的评论

Guillaume 2019-7-23

在 MATLAB Online 中打开

The longer the vectors are the slower the second and third iterations become

Note that the following is all (informed) suppositions since the internals of fwrite are not documented. I assume matlab fwrite delegates to C or similar fwrite.

Writes are typically buffered (a standard buffer size used to be 2kB, not sure if it's still the case). If the data you write is less than the buffer size, then your skipped writes can all be done in the buffer in memory. Hence it's fast. However, once you've past the buffer size, the buffer needs to be written to the disk while doing a write and needs to be reread for the next write. So we have 2 scenarios:

numel(i1) + numel(i2) + numel(i3) < size(buffer):
fwrite(fid, i1, 'int16', 4)
    -> create buffer
    -> fill buffer with 0s and i1
fseek(fid, 2, 'bof')
    -> rewind pointer to element 2 of buffer
fwrite(fid, i2, 'int16', 4)
    -> fill existing buffer
etc..

numel(i1) + numel(i2) + numel(i3) > size(buffer):
fwrite(fid, i1, 'int16', 4)
    -> create buffer
    -> fill buffer with 0s and 1st part of i1
    -> write buffer to file
    -> clear buffer
    -> fill buffer with 0s and next part of i1
    -> until all i1 has been written
fseek(fid, 2, 'bof')
    -> rewind pointer
fwrite(fid, i2, 'int16', 4)
    -> read 1st part of file into buffer
    -> write 1st part of i2 into buffer
    -> write buffer to file
    -> read next part of file into buffer
    -> write next part of i2 into buffer
    -> write buffer to file
    -> etc until all i2 has been written
etc...

It's that or the data is not buffered at all for i2 and i3 and each byte has to be written directly to the file skipping around which would also have massive performance issues.

Again, the way to do this properly is to read the data in chunks, perform the interlacing in memory and write it as a contiguous chunk. That's how you'd do it with a low-level language like C.

Guillaume 2019-7-23

编辑：Guillaume 2019-7-23

在 MATLAB Online 中打开

No, if you're reading and writing in chunk, the buffer size doesn't matter. and for best performance, I would think you'd want to be above that buffer size. The chunk size should only depend on how much memory you want to use / have available.

So the code would be something like:

%filenames: cell array of input files
chunksize = 4000;  %whatever you want, the actual size in bytes will be nfiles * chunksize * sizeof(datatype)
buffer = zeros(numel(filemames), chunksize, 'int16');
fidin = zeros(size(filenames));
numread = zeros(size(filenames));
%open the input files
for fileidx = 1:numel(filenames)
    fidin(fileidx) = fopen(fullfile(datadir, filenames{fileidx}), 'r');
    assert(fidin(fileidx) > 0, 'failure to open file %d', fileidx);
end
fidout = fopen(newfile, 'w');
while true
    for fileidx = 1:numel(filenames)
        [bytes, numread(fileidx)] = fread(fidin(fileidx), chunksize, 'int16');
        buffer(fileidx, 1:numread) = bytes;
    end
    assert(all(numread == numread(1)), 'files don''t have the same length');
    fwrite(fidout, buffer, 'int16');  %since data has been read in rows and write is by column, the data is written interlaced
    if numread(1) < chunksize || feof(fidin(1))
        break;
    end
end
fclose(fidout);
for fileidx = 1:numel(filenames)
    fclose(fidin(fidin));
end

Kouichi C. Nakamura 2019-7-23

Thanks. This is very helpful!

请先登录，再进行评论。

Answer 2

Walter Roberson 2019-7-22

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/472847-why-fwrite-is-320x-slower-in-the-second-interation-and-onwards-when-writing-interleaved-data#answer_384243

编辑：Walter Roberson 2019-7-22

fseek after end of file followed by fwrite, results in the data being written at eof. This is contrary to POSIX which requires that the gap be filled with 0 (possibly implicitly with a Demand Zero scheme). In MATLAB if you want to write at some point after eof you must write into the gap yourself.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Kouichi C. Nakamura 2019-7-23

在 MATLAB Online 中打开

Thanks for your reply.

> In MATLAB if you want to write at some point after eof you must write into the gap yourself.

In my example above, there are four iterations.

For i = 1, fwrite reaches the end of the file for the first time.

For i = 2 to 4, however, fwrite needs to add 2 extra bytes at the end of file for every iteration. But does this small addition takes 320x more time? Or have I completely missed your point here? I don't really know what POSIX means to be honest.

In oder to further examine the issue, I wrote a simplified test code, which I thought inherits all the key features of the original code. To my dismay, this didn't result in massive slow down. Mmm, I tested this on Mac, while the above was done on Windows.

i1 = ones(1000,1,'int16');
i2 = ones(1000,1,'int16').*2;
i3 = ones(1000,1,'int16').*3;
fid  = fopen('temp.bin','w')
t1= datetime;
fwrite(fid,i1,'int16',2*2);
t2 = datetime;
t3(1,1) = duration(t2-t1);
t1= datetime;
fseek(fid,2, 'bof')
fwrite(fid,i2,'int16',2*2);
t2 = datetime;
t3(2,1) = duration(t2-t1);
t1= datetime;
fseek(fid,2*2, 'bof')
fwrite(fid,i3,'int16',2*2);
t2 = datetime;
t3(3,1) = duration(t2-t1);
fclose(fid);
t3.Format = 'dd:hh:mm:ss.SSSS'

t3 = 3×1 duration array

00:00:00.0665

00:00:00.0738

00:00:00.1156

请先登录，再进行评论。

Why fwrite is ~320x slower in the second interation and onwards when writing interleaved data?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

7 个评论
显示 5更早的评论隐藏 5更早的评论

更多回答（1 个）

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

Why fwrite is ~320x slower in the second interation and onwards when writing interleaved data?

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

7 个评论 显示 5更早的评论隐藏 5更早的评论

更多回答（1 个）

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

7 个评论
显示 5更早的评论隐藏 5更早的评论

1 个评论
显示 -1更早的评论隐藏 -1更早的评论