Hello,
I am using a local Matlab client (r2011b) connected to our remote cluster with Torque/Moabas the resource manager/scheduler. I can successfully submit and run jobs using the generic scheduler interface, which is great, especially after reading the helpful example files for pbs in a nonshared file system included in the toolbox.
However, when I'm submitting alot of jobs [hundreds] all with very small data sets, say, 10k each, the file mirroring seems to bog my Matlab client down and keep it in a near perpetual 'busy' state. It is also very, very slow, which I can't understand why given the small amount of data and results. Maybe it has trouble mirroring at a high level with lots of files?
Essentially, I have a script that does some preprocessing and then goes through and submits all my jobs. When that's done, I close the Matlab client (since they will be running for a while!) When I reopen it, I set up the scheduler like usual:
(where "torque_scheduler" is a script I modified from the Matlab example to return the interface) and then it immediately goes "busy" and just sits there. I can see it updating files in my working folder by refreshing the explorer window and checking timestamps, but it is INCREDIBLY slow-- it will be stuck doing this for hours.
My question is-- is there some way that I can manually shut off file mirroring so that it doesn't get so slow and then copy the data myself when the jobs or done, or wait until I do a 'findJob' command? When it hits the crazy busy state, it becomes unresponsive for hours. And also of note: when I submit these jobs, my Matlab client begins usurping VERY large amounts of memory-- it was up to 8 GB at one point towards the end of my submission routine... what's up with that? Is that all the zipping/sending it's doing?
One thing that makes me wary of the file mirroring is that I keep getting errors like this every so often:
_Error using distcomp.genericscheduler/pSubmitJobCommon (line 64)
Job submission did not occur because the user supplied SubmitFcn (parallelSubmitFcn) errored.
Error using parallel.cluster.RemoteClusterAccess/startMirrorForJob (line 364)
Failed to start mirror for job with ID 20.
Error in parallelSubmitFcn (line 132)
remoteConnection.startMirrorForJob(job);
Error in distcomp.genericscheduler/pSubmitJobCommon (line 48)
feval(submitFcn, scheduler, job, setprop, args{:});
Error in distcomp.genericscheduler/pSubmitParallelJob (line 24)
scheduler.pSubmitJobCommon( job, scheduler.ParallelSubmitFcn );
Error in distcomp.simpleparalleljob/submit (line 47)
scheduler.pSubmitParallelJob(job);
Error in getsubnetworks (line 44)
submit(job);
Caused by:
Error using parallel.cluster.RemoteClusterAccess/waitForChoreToFinishOrError (line 919)_
which makes me think I should just turn it off. Seems great for small workloads or small amounts of data, but it seems to get very overwhelmed when you throw alot of jobs at it. Any suggestions? Is there an easy way to turn this off and do it manually that anybody has had success with?
Thanks-- Nick
PS: Is there any way to add some kind of taskbar or progress bar so that I can see what Matlab is doing in the background during it's massive 'busy' binge?
