I have a script which consists of a main loop and subfunctions. In order to speed up the calculation in the main loop, I use parpool('threads') and parfeval to calculate the output of the subfunctions ahead of time, so that when it reaches the right iteration, the subfunctions already completed and I can just retrieve the output. This is working well, however, I have a machine with limited memory, so I could run into out-of-memory kill if I'm not careful.
I would like to be able to manually control when to start a job, is it possible? For example, I set up parpool with 3 workers using maxNumCompThreads, and then create 10 jobs. Job 1-3 will start running and 4-10 will be in queue. The moment job 1 finishes, job 4 will start. At this moment, let's say I'm not ready to read the output of job 1, I will then need the memory to store the output of job 1-4 potentially job 5 too, which can become overwhelming for my machine. So I'd like to be able to start job 4 only after I do fetchOutputs(job1) and delete(job1).
Thanks.
