The issue you'll have with parfor is that it can't terminate early. I believe what you're suggesting is to do checkpointing -- having MATLAB write to a file(s) sporadically. MATLAB doesn't have this, but looks like you're trying to explicitly do this. This can be useful to requeue the job and start further downstream or to have a minimal set of output before a job bails, etc.
maxruntime=hours(4); %set maximum runtime to 4 hours
starttime=datetime; %save the time at which we start the parfor
for packets = 1:N
f(packets,1) = parfeval(@unit_of_work, ..);
end
for packets = 1:N
[idx, ..] = f(packets).fetchNext();
if (datetime-starttime)<maxruntime
continue
else
% About to run out of time, cancel all futures
f.cancel
end
end
%code to write out the accumulated arrays from the parfor loop is here
Looks like Rik has an improvement with now vs datetime you could use as well.
Obviously, this is a different approach then parfor, but it gives you the flexibility to end the loop early. You'll need to aggregate your results from fetchNext.
Another thought is to use DataQueue. doSomething will very quite a bit, depending on what you want to do with your aggregated array, but it's a starting point.
q = parallel.pool.DataQueue;
afterEach(q, @doSomthing);
parfor packets=1:N
if (datetime-starttime)<maxruntime
%the normal loop code goes in here
%arrays are accumulated
q.send(<the variable you want to accumulate>)
end
end
function doSomething(D)
% do something with D (e.g, write to file, etc.)
end
