Parfor with user made classes

11 次查看（过去 30 天）
Adrian
Adrian 2023-3-1
评论： Adrian 2023-5-8
Hello.
I am using a user made toolbox in my current project and with that user made classes. Now I have two for-loops which by its defined purpose will always have to deal with larger data amounts, where the first one allocates and the second one computes. As an example, the first loop looks just like
for i = 1 : els
switch etype(strct(i))
case 'sref'
% Do something
case 'aref'
% Do something
case {'boundary', 'path'}
% Do something
otherwise
% Do something drastic
end
end
etype is also part of the user made library and returns some of the names in the cases. There have been no issues with the library. The code works within for-loops.
Now considering the data amount will always be large and there has to be pre-allocation in some way, I wanted to try to decrease runtime by using parfor, since the do somethings should be separable. But if I exchange the for with parfor, I get the following warnings:
Warning: Element(s) of class 'gds_element' do not match the current constructor definition. The element(s) have been converted to structures.
> In parallel.internal.pool.deserialize (line 33)
In remoteParallelFunction (line 66)
Warning: Element(s) of class 'gds_element' do not match the current constructor definition. The element(s) have been converted to structures.
Warning: Element(s) of class 'gds_element' do not match the current constructor definition. The element(s) have been converted to structures.
Analyzing and transferring files to the workers ...done.
Warning: Element(s) of class 'gds_element' do not match the current constructor definition. The element(s) have been converted to structures.
> In parallel.internal.pool.deserialize (line 33)
In remoteParallelFunction (line 66)
Warning: Element(s) of class 'gds_element' do not match the current constructor definition. The element(s) have been converted to structures.
which culminates in the following error
Error using downstream
The source code [url] for the parfor-loop that is trying to execute on the worker could not be found.
Caused by:
Undefined function 'etype' for input arguments of type 'cell'.
Error using remoteParallelFunction
Worker unable to find file.
Undefined function 'etype' for input arguments of type 'cell'.
Is it the case that Matlab just can't handle any user defined classes in parallel loops? Is that a general thing? Or did I make a mistake? Is it because of the Toolbox?
I do understand that etype can't deal with cells - an annoyance of the toolbox - but I'm not passing a cell but specifically a gds_element. Even a workaround a la
elem = strct(i);
if iscell(elem); elem = elem{1}; end
switch etype(elem)
didn't help.
  2 个评论
Steven Lord
Steven Lord 2023-3-1
Where is the code you wrote in your first code block located?
  • Is it in a script or function outside all of the classes you're trying to use?
  • Is it in a method of one or more of those classes (and if so what kind of method: ordinary, constructor, static, something else?)
Are those classes part of a heterogeneous class hierarchy?
Are you creating the objects before the parfor loop and using them inside the loop or does your code inside the loop attempt to create the objects?
Adrian
Adrian 2023-3-3
The external toolbox lives in a subfolder of my Matlab folder, where all my external toolboxes live. This particular block is in a function in the respective project subfolder of my Matlab folder. So it is the first option, in a function outside the class.
Nothing derives from matlab.mixin.Heterogeneous or for fact anything at all, so I would say no it is not a heterogeneous class hierarchy.
All the objects are created before the for-loops before being acted on. The first loop is for pre-allocation, hence there are only bits of three arrays being flipped. Under no circumstance can the same bit be flipped twice.
The second loop is a bit more difficult to linearize. If we get the first one running, I'll be happy to dissect the second one in more detail. But neither creates objects, they all just act upon already created objects.
I tried just changing the first loop to parfor, but got the same error.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

Raghav
Raghav 2023-5-4
Hi,
Based on the question, it can be understand that you are facing compatibility issues of user made classes with Parallel computing toolbox.
The warnings and error message you are seeing suggest that there might be compatibility issues with the user-defined classes and the parallel computing toolbox in MATLAB. When using parfor in MATLAB, the data that is being passed between the worker processes needs to be serialized, sent across to the worker processes, and then deserialized. This process can result in issues with custom classes that are not serializable, or whose definitions are not available on the worker processes.
It's possible that the gds_element class is not serializable or not defined on the worker processes, which is causing the errors you're seeing. You may need to modify the implementation of your gds_element class to ensure that it is compatible with the parallel computing toolbox.
One thing you could try is to modify your code to pass only the necessary data that is needed in the loop to the worker processes, rather than passing the entire gds_element object. For example, you could extract the relevant data from the gds_element object and create a new struct that only contains the necessary data. This could help to reduce the size of the data being passed between the worker processes and potentially improve performance.
Alternatively, you could try using the spmd construct instead of parfor, which is designed for parallel computing with MATLAB. spmd allows you to execute code on multiple MATLAB workers and explicitly control the data that is passed between the workers. This could be a better option if you are dealing with custom classes or data structures that are not compatible with parfor.
The official documentation for spmd is mentioned below:
https://www.mathworks.com/help/parallel-computing/spmd.html
Hope it helps,
Raghav Bansal
  1 个评论
Adrian
Adrian 2023-5-8
Hello.
The project is done, so I don't want to re-open that can of worms and try spmd, since it is working right now, just non-parallelized.
And yes, there are apparently compatibility issues. There are also some other related topics and error descriptions online. Probably gds_element needs to be serialized. But how can I check if something is serializable? What are the conditions that need to be satisfied? And why is the error description so utterly useless? What is the current constructor definition? The gds_element is a really easy class, so where lies the problem? There is just no good information online.
Whatever, I solved my problem some time ago.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

标签

产品


版本

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by