Parfor loop with mex-file call crashes all workers on one computer, but runs fine on others

Question

Bernt Nilsson 2019-7-4

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/470275-parfor-loop-with-mex-file-call-crashes-all-workers-on-one-computer-but-runs-fine-on-others

评论： Jan 2019-12-4

Hello!

We have a code which runs either in serial or parallel mode. In the part under inspection one of our mex-files is run on either the complete set of data (serial operation) or on a part of the data consistent with the current number of workers (parallel operation). When run in serial mode the code works fine, but when run in parallel mode on a 6-core computer (with 2, 4 or 6 workers) the workers crash with messages like:

Warning: A worker aborted during execution of the parfor loop. The parfor loop will now run again on the remaining
workers. 
> In distcomp.remoteparfor/handleIntervalErrorResult (line 240)
  In distcomp.remoteparfor/getCompleteIntervals (line 387)
  In parallel_function>distributed_execution (line 745)
  In parallel_function (line 577)
  In foo3 (line 138)
  In foo2 (line 142)
  In foo1 (line 61)
  In foo (line 166)

A little later sometimes we get:

Error using distcomp.remoteparfor/rebuildParforController (line 194)
All workers aborted during execution of the parfor loop.
Error in distcomp.remoteparfor/handleIntervalErrorResult (line 253)
                obj.rebuildParforController();
Error in distcomp.remoteparfor/getCompleteIntervals (line 387)
                            [r, err] = obj.handleIntervalErrorResult(r);
...
The client lost connection to worker 3. This might be due to network problems, or the interactive communicating job might
have errored.
 
Warning: 4 worker(s) crashed while executing code in the current parallel pool. MATLAB will attempt to run the code again
on the remaining workers of the pool. View the crash dump files to determine what caused the workers to crash. 

The crash dumps don't say a lot, but conclude with:

This error was detected while a MEX-file was running. If the MEX-file
is not an official MathWorks function, please examine its source code
for errors. Please consult the External Interfaces Guide for information
on debugging MEX-files.

When run on three other computers the code works fine, in both serial and parallel mode. Two computers with 6-core CPU:s and a notebook with a 2-core CPU. It is possible to create different size pools, and the resulting output is always as expected. I have tried the code on all computers using the same number of workers (4) where possible.

This leads me to believe the mex-file is correct.

I am at a loss concerning what to try next, and would appreciate any hints on how to move forward.

7 个评论
显示 5更早的评论隐藏 5更早的评论

Jan 2019-7-5

The only hint, that te problem occurs in a specific MEX funtion is:

"This error was detected while a MEX-file was running."

Do I understand correctly: The code runs fine when running with 1 worker on all machines. In parallel mode, it works correctly on 2 computers, but crashes reproducible on 1 computer.

It would be valuable to find out, where the MEX function is failing. Perhaps it calls anoter library function, which is installed in a thread-safe version on the 3 computers, but on the failing computer the version is not thread-safe. Perhaps the C-code (if it is C - you did not mention any details about the MEX function yet) uses an uninitialized pointer. Then the function will crash "randomly", but this depends e.g. on the size of the installed memory also.

Then a compiled function runs successfully on some computers is not a proof, that it is "correct". Do you remember on how many computers MS Windows is running? But you still get a Blue Screen of Death sometimes.

I suggest to post the relevant part of the source code.

Bernt Nilsson 2019-7-5

在 MATLAB Online 中打开

Thank you both for taking the time to look at my problem.

I didn't mention this in the first post, but the problem machine is actually two, identical, machines, and the results are the same on both, i.e. the code will not run in parallel mode on either. The three other machines run the code fine. All machines run R2019a rev 3, but the ones where it doesn't work use a different license, still with Parallel Computing Tb of course.

A very simple overview of the code is:

main.m:

...
if serial
    output = foo_serial(input,...);
else
    output = foo_parallel(input,...);
end
...

foo_serial.m:

...
output = mex_file(input);
...

foo_parallel.m:

...
% Slice Input data
M = N/numWorkers;
input_slc = zeros(M,numWorkers);
output_slc = zeros(M,numWorkers);
% Populate input_slc
...
% Run in parallel
parfor k = 1:numWorkers
    output_slc(:,k) = mex_file(input_slc(:,k));
end
...
% Assemble output from slices
...

To sum up the situation: Two identical computers using Windows 10 run fine in serial mode, but when run in parallel mode all workers crash. I have tried 1, 2, 4 and 6 workers.

Three other computers, two 6-core ones running Windows 10 and openSUSE Linux and a 2-core laptop running Windows 10, run fine in both serial and parallel versions. Here I have tried 2, 4 and 6 workers, except on the two-core laptop which will only run 2 workers.

I have recompiled the mex-file on the troubled PC:s, but the results are still the same. And I actually did run a diagnostic for about 24 hours on one of the "failing" computers, but without errors. Not memtest86, but a diagnostic routine (part of which stress test memory) from HP.

The mex-file is built from 30+ C and FORTRAN routines. I didn't write any of them, so it would be difficult for me to debug them (which isn't to say I shouldn't). I said that I "believe" it is correct, but there could of course still be errors in it. And there is nothing random about the execution on the troubled PC:s, the workers crash every time.

From the crash dump files I get that the problem is "Access violation", which would suggest the problem is with the mex-file. But this access violation only happens in parallel use of the same mex-file, and only on two out of five computers.

Is there a way to debug the differences between the computers with and without problems? For example, list all (or any) dynamic link libraries used, and see if there are differences?

Christopher Grose 2019-12-3

编辑：Christopher Grose 2019-12-3

在 MATLAB Online 中打开

I'm having this problem as well.

Could it occur because of uninitialized variables in the mex C function (I don't think i have any uninitialized pointers)?

For example, my C codes look something like

    void mexFunction( int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
    {
     /* DECLARATIONS, INPUTS */
        double    *Gphase        =       mxGetPr(prhs[0]);
        double    *dGdC          =       mxGetPr(prhs[2]);
        double    *StrainEM      =       mxGetPr(prhs[4]);
        double     R             =   mxGetScalar(prhs[5]);
        int32_t    p,c,ci,j;
        for (j=0; j<1000; j++) {
            somefunc(Gphase,dGdC,StrainEM,R,p,c,ci);
        }
    }

Could p,c,ci,j variables be screwing things up?

On a 28 core CPU I quickly lose half my workers, followed by a random but gradual elimination of further workers.

I get the error

    Warning: A worker aborted during execution of the parfor loop. The parfor loop will now run again on the remaining workers. 

and the worker that takes over complete the task without problems, and the functions are all fully deterministic, so the workers are just dying for some other problem

Jan 2019-12-4

It depends on what happens inside somefunc(). The output of mxGetPr() should be treated as const pointer, so do you modify the contents? p, c, and ci are declared, but not initialized - do you use them correctly? Maybe "something like" conceals the actual problem. Please post the relevant part of the real code.

请先登录，再进行评论。

请先登录，再回答此问题。

Parfor loop with mex-file call crashes all workers on one computer, but runs fine on others

7 个评论
显示 5更早的评论隐藏 5更早的评论

回答（0 个）

另请参阅

类别

标签

产品

Community Treasure Hunt

Parfor loop with mex-file call crashes all workers on one computer, but runs fine on others

7 个评论 显示 5更早的评论隐藏 5更早的评论

回答（0 个）

另请参阅

类别

标签

产品

Community Treasure Hunt

7 个评论
显示 5更早的评论隐藏 5更早的评论