Using parfor loop to restructure large dataset

Question

Mackenzie Dughi 2021-3-9

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/767797-using-parfor-loop-to-restructure-large-dataset

回答： Alvaro 2023-1-19

I'm working on restructuring particle trajectory data to be used in our analysis. This is a 24Gb file that contains the particle ID and position, time and status. The structure currently looks like the following:

I would like to restructure the dataset so that I have a "traces" structure that has "coos" cells that are indexed by the particle ID. In these cells I want to put each tracjectorie's PositionX, PositionY, PositionZ, Time, and Status.

I wanted to use a parfor loop to cut down on computational time but I keep running into the issue of broadcast variables in the following code:

numID = length(unique(pt.ID));
coos = cell(height(1),width(numID));
parfor i = 1:numID
   indx = find(pt.ID(:)==i);
   x = pt.PositionX(indx); y = pt.PositionY(indx); z = pt.PositionZ(indx);
   t = pt.Time(indx); status = pt.Status(indx);
   coos{i}(:,1) = x;
   coos{i}(:,2) = y;
   coos{i}(:,3) = z;
   coos{i}(:,4) = t;
   coos{i}(:,5) = status;
   
end

where the first line within the parfor loop "pt.ID" is a broadcast variable. If I understand correctly, broadcast variables are in which all workers need access to and therefore hindering computational time. Is there any kind of workaround that allows me to find the indices of each trajectory and populate my traces structure without running into this issue? Additionally, since I'm dealing with such a large dataset, is there anyway to increase the 2Gb array limit that can be passed into parfor?

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Mohammad Sami 2021-3-10

在 MATLAB Online 中打开

I dont think it will really speed up the restructuring. Doing it this way will likely create a lot of overhead for moving data between the processes.

Also you dont need to call the find function in the following. Logical indexing should be enough.

indx = pt.ID==i;

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Alvaro 2023-1-19

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/767797-using-parfor-loop-to-restructure-large-dataset#answer_1151985

I agree with Mohammad that I would not expect parallelization to significantly speed up the rearranging of this struct since you are broadcasting not just pt.ID but the entire pt struct to each worker.

If you still wish to use parallelization in a more efficient manner, you might be able to extract the data from your struct and slice it so that each worker only operates with a subset of the data.

https://www.mathworks.com/help/parallel-computing/sliced-variable.html

Alternatively, if you want to keep your struct intact, you could write a function that performs that manipulations on the struct and pass it to structfun which supports ThreadPool. Then use num2cell on the resulting array.

https://www.mathworks.com/help/matlab/ref/structfun.html

If the problem is that you want to use MATLAB while rearranging that struct, then you could consider a batch job in your local computer or you could send it to a cluster if you have access to one.

https://www.mathworks.com/help/parallel-computing/simple-batch-processing.html

Also, upgrading from MATLAB R2013a should remove the 2 gb limit transfer for broadcast variables.

https://www.mathworks.com/matlabcentral/answers/34306-parfor-attempt-to-serialize-data-which-is-too-large#answer_114553