Hi Shane,
There might be several options here, in no particular order
1. For starters, you might consider starting the pool (would it be local?) as part of the startup process of your application. By default, the parallel pool will shutdown after a timeout period. You might want to suggest they disable this, so that it doesn't shutdown in case they don't use it for a while. Of course this may have some performance degregation if you don't need the pool running at all. When your application is done, and if you've had them override auto-shutdown of the pool, you might want to close the pool as well.
2. In R2020a we introduced "threads" pool, which might be quicker to start. You can read more here
p = parpool("threads");
parfor fidx = 1:numel(files)
process(files{fidx})
end
3. parfor takes several arguments. Two to look at are:
- Specifiying the maximum number of workers to use. For example
if nfiles>=threashold
psize = 4;
else
% Not enough files to process in parallel
psize = 0;
end
parfor (fidx = 1:numel(files),psize)
process(files{fidx})
end
If psize is 0, then parfor operates as a for (though maybe a bitter slower? worth checking).
- Request a cluster pool (introduced in R2019a), rather than a parallel pool. By specifying a parfor options, jobs are submitted via batch submission. There's more to understand about cluster pools, so read more here
The advantage to this is that it doesn't take as long to start the pool. This is more applicable to jobs running on an cluster.
cluster = parcluster;
opts = parforOptions(cluster);
parfor (fidx = 1:numel(files),opts)
process(files{fidx})
end
Raymond
