Batch scheduler or pick up the run after wall time expires or checkpoints

1 次查看（过去 30 天）
nlm
nlm 2021-1-26
评论： Jan 2021-1-26
Hi,
I have a matlab program which I submitted as sbatch file. However, the walltime for matlab license on supercomputer to which I submitted expires after 24hrs. My data is saved at each step so I am not worried of losing the data at any point. I am trying to find a way to start running the file after it left off on the for loop. For example, considering I am running a time program like, if my code ends at j = 45000 and j = 30000 after 24hrs, how should I restart again at i = 45001 and j = 30001. Any hints ?
for i= 1:100000
for j = 1:60000
code....
save(data)
end
end

请先登录，再回答此问题。

回答（1 个）

Jan
Jan 2021-1-26
编辑：Jan 2021-1-26
DataFile = 'C:\Temp\YourData.mat';
if isfile(DataFile)
Data = load(DataFile);
else
Data.i0 = 1;
Data.j0 = 1;
end
for i = Data.i0:100000
Data.i0 = i;
for j = Data.j0:60000
Data.j0 = j;
Data.value = code...
save(DataFile, 'Data');
end
end
Saving the file 6'000'000'000 times will waste a lot of time. Then this will loose the last inner loop, but is most likely much faster:
DataFile = 'C:\Temp\YourData.mat';
if isfile(DataFile)
Data = load(DataFile);
else
Data.i0 = 1;
end
for i = Data.i0:100000
Data.i0 = i;
for j = 1:60000
Data.value = code...
end
save(DataFile, 'Data');
end
A smart option would to store the file every minute only using clock and etime. Then you loose less then a minute per day, but need only the time for 1440 save commands.
  2 个评论
nlm
nlm 2021-1-26
编辑：nlm 2021-1-26
There is no pre-allocation of memory like data.mat. I just writing the data into .bin files. Can you suggest something for my code below ?Thanks.
%%%%%%% NDVI Processing %%%%%%%%%%%%%%
%%%%% Code by Maheshwari Neelam %%%%%% Date: 12/07/2020
clear all
close all
clc
addpath('path')
filename = 'path/ocean_mask.mat';
mask = matfile(filename);
D=dir('./20*');
Nobs=size(D,1);
date_fixed = (1:1:366)' ;
num_row = 1000; % rows per group
% Initialize day counter
clear time
k=1;
band_ID = 1 ;
lat_band = (1:num_row)+(band_ID-1)*num_row;
ND = 366;
tmp = 'path/data/';
bnd = '/data_lat10.bin';
fids = zeros(1,ND);
for day = 1 : ND
fids(day) = fopen(sprintf('%s%d%s', tmp, day, bnd), 'a');
end
tic
for lat = 1:length(lat_band)
tic
for lon = 1:36000
if (mask.ocean_mask(lat_band(lat),lon)~=9)
data_1km = code;
end
else
data_1km = ones(366,1)*9;
end
for day = 1:ND
fwrite(fids(day), data_1km(day), 'double');
end
clear data_1km day
end
toc
end
toc
fclose(fids)
Jan
Jan 2021-1-26
Removing the data by the clear commands is usually a waste of time in Matlab.
Adding a folder to the path only to access the local files is a bad design. Only Matlab's functions should be part of the path, but there is no reason to expand the path for accessing data files. Use absolute pathnames instead.
I cannot run your code, but it looks like the part "mask.ocean_mask(lat_band(lat),lon)" might consume a remarkable part of the processing time. Do you have good reasons to access the MAT file extremely often? Is it far too large to match into the RAM?
"There is no pre-allocation of memory like data.mat." - I've showed you a method how the loop indices can be stored such, that restarting the script starts the loops at where they have been stopped bevor. This is not a pre-allocation. Nevertheless, this method cannot work reliably in your case, because you append data to open files. If the schedular of the cloud system cancels your process brutally, there is no guaratee, that the open files are left in an accurate state. There is no way to modify your code to let it be stoppable.
As far as I can see, you need a completely different approach. The idea of opening 366 file simultaneously and the time consuming access of the MAT file might be the main problem. If "code" does not hide any huge computations, the code should run fluently in far less then 24 hours. So instead of changing the code to accept a hard stop after 24 hours computing time, it is much better to improve the code to run efficiently. Currently it seems to spend the time with stressing the disk only.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Data Import and Analysis 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by