Fast code to open and write separate 366 .bin files for each (row,col)

1 次查看（过去 30 天）
nlm
nlm 2021-1-24
评论： nlm 2021-1-25
I am trying to speed up my code. I have a 18000*36000*366 matrix. I split the matrix into 18 bands of each 1000* 36000 .bin files located in 366 folders. I plan to run the same code for 18 times. The problem is it takes 3.5hrs to process 1*36000 like I wanted. It is super slow, any help is appreciated. Below is my code,
for lat = 1:length(lat_band) % length of lat_band = 1000
for lon = 1:36000
if (mask(lat,lon)~=9) % if ocean then skip
data_1km = data; % matrix of size [366,1];
else
data_1km = NaN(366,1);
end
tmp = 'path';
tic
for day = 1:366
fwrite(fopen(sprintf('%s%d', tmp, day, '/band_1.bin'), 'a'), data_1km(day),'double');
end
fclose('all');
clear data_1km day
toc
end
end

请先登录，再回答此问题。

采纳的回答

Walter Roberson
Walter Roberson 2021-1-24
That code fopen()'s 366 files inside the loop, which can consume all of the file handles. Better would be
for day = 1:366
fid = fopen(sprintf('%s%d', tmp, day, '/band_1.bin');
fwrite(fid, 'a'), data_1km(day),'double');
fclose(fid)
end
If your system can handle and your process is authorized to have 366 files, then fopen() all of them before for lat and index into the list of handles in the loop. You are doing a lot of fopen() of the same file, and that is expensive.
  11 个评论
显示 9更早的评论
Walter Roberson
Walter Roberson 2021-1-24
I will run this code for 18 lat_bands.
What is the file name to be used for band #4 day #7 ? And could you confirm that the variable lat is the one that stores the current lat band number?
nlm
nlm 2021-1-25
Can you tell me why running a code as simple as 0.012 secs. That is a long time isn't it ?
data = NaN(366,1);
tic
for day = 1:ND
fwrite(fids(day), data(day), 'double');
end
toc

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Low-Level File I/O 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by