I have noticed the following behavior of the parpool-function in Matlab (2023a). When I use something like
pool = parpool('Processes', 128)
it return with an error
Starting parallel pool (parpool) using the 'Processes' profile ...
Parallel pool using the 'Processes' profile is shutting down.
Error using parpool
Parallel pool failed to start with the following error. For more detailed information, validate the profile 'Processes'
in the Cluster Profile Manager.
Caused by:
Error using parallel.internal.pool.AbstractInteractiveClient>iThrowWithCause
Failed to initialize the interactive session.
Error using parallel.internal.pool.AbstractInteractiveClient>iThrowIfBadParallelJobStatus
The interactive communicating job failed with no message.
This error is due to the fact, that on the used Linux server, the number of files is limited
and it seems that Matlab is opening >1000 files per worker requested for the pool. This seems to be an unreasonable large number of files imho so my question is if I can somehow prevent this?
I know that I can of course increase the limit on the number of files in the OS. I also noticed that
pool = parpool('Threads', 128)
does not suffer from this problem (and the pool has also a much short startup time), but in parfor-loop a piece of Mex-compiled code is used and this is not possible for a pool of threads:
Use of MEX functions is not supported on a thread-based worker.
So an alternative question would be how to use a MEX function on a thread-based worker?
