parfor results core dump

10 Comments

ShowHide 7 older comments

OCDER on 16 Nov 2017

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_505630

Link

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_505630

If this works with a regular for loop, but not parfor, it's likely a concurrency issue. If element is an object and multiple workers are trying to split, modify, and reassemble an object, then maybe it crashes.

What is the object array element (what is the code that defines this object, or what does it store?)

What does element(i).stiffnessMatrix do to all these objects?

Do you have a simple full set of code for us to test that for loop?

Zoltán Csáti on 17 Nov 2017

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_505993

⋮

Link

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_505993

Thanks for the reply. For several days I have been trying to locate where the error comes from. Here is what I experienced:

1) The code fails at random locations but always in loops (let them be for loops or parfor loops)

2) I use third-party mex functions but they do their job correctly and I never get error messages after invoking them.

3) The segmentation fault only occurs for larger problem (i.e. when larger object arrays are allocated. Being "large problem" is relative: on Windows it allows more, on Linux it fails earlier.

Since the code is quite large (contains several dozens of functions), I didn't attach it. Unfortunately, I couldn't produce a minimum working example because, as I wrote, the code fails at different locations.

OCDER on 17 Nov 2017

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506100

⋮

Link

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506100

I would suspect the 3rd party MEX functions has a bug that manifests only when processing large data. You may have to debug the MEX source code directly as segmentation faults indicate a pointer error in the C code. Finding this bug would be VERY hard, especially since the code works sometime and then fails randomly, and for parfor and for loops...

You could try to pinpoint the inputs that cause the crash by using a regular for loop and saving variables after each successful loop. Once it crashes, restart from the prior loop and see if the next loop crashes again. If you can replicate the crash conditions, then debugging just might be possible.

Zoltán Csáti on 17 Nov 2017

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506129

⋮

Link

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506129

Could you please detail it?

Do you mean, for each for loop I should output the loop index so as to see until which index it succeeds?

OCDER on 17 Nov 2017

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506131

⋮

Link

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506131

Yes, something like that. For instance:

values = cell(1, nElement);
for i = 1:nElement
   save('temp.mat');
   values{i} = element(i).stiffnessMatrix;
   mexFunction2(...)
   mexFunction3(...)
end

If it crashes always on the 79th iteration, then you can load 'temp.mat' and run the remain codes in the loop line-by-line until you find the mex function causing the crash.

Zoltán Csáti on 17 Nov 2017

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506134

⋮

Link

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506134

Thank you. But the problem is that when I call mexFunction2 in a loop it does not fail. Later, when I do not even call a mex function in a for loop, that's when the program crashes.

OCDER on 17 Nov 2017

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506138

⋮

Link

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506138

Without using any mex functions, does the loop at least crash at the same iteration number?

Zoltán Csáti on 17 Nov 2017

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506141

⋮

Link

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506141

I cannot test it without the mex function because I need that for the later codes to work. And even later, the loop without the mex function crashes at different iteration numbers. And different parts of the code on Linux and on Windows.

OCDER on 17 Nov 2017

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506172

⋮

Link

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506172

This does sound like one of the worst case scenario for debugging codes. You might have to contact the authors of the 3rd party MEX codes to find the memory allocation bug. Without looking at all the codes, it'll be hard to pinpoint the issue. This might help with debugging strategies:

http://www.cs.cornell.edu/courses/cs312/2006fa/lectures/lec26.html

Zoltán Csáti on 18 Nov 2017

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506318

⋮

Link

Direct link to this comment

https://ww2.mathworks.cn/matlabcentral/answers/365665-parfor-results-core-dump#comment_506318

Thank you. I may use valgrind/callgrid to profile the mex file. If you write an answer I will accept it. Although my problem persists (it's too vague to be solved immediately), you gave me good advices.

You are now following this question

10 Comments

Answers (0)

See Also

Categories

Tags

Products

Community Treasure Hunt

Tags

Products

See Also

Community Treasure Hunt

You are now following this question

parfor results core dump

10 Comments

Answers (0)

See Also

Categories

Tags

Products

Community Treasure Hunt

Tags

Products

See Also

Community Treasure Hunt

An Error Occurred

WeChat