Parfor loop: how to keep the temporary variables ?

15 次查看(过去 30 天)
Hello,
I am working with a time-consuming code and I want to put in parallel computing the independant functions I use.
I have the following script (only a fraction of it):
parfor i = 1:2
if i == 1
[T20cm_land, albedo_land, smb_land, swe_land, T1m_land, T0_land, abl_land,...
acc_land, runoff_land] = display_output_land(point_output_path_land, yr, mm, dd, date);
else
[T20cm_glacier, albedo_glacier, smb_glacier, swe_glacier, T1m_glacier, T0_glacier,...
abl_glacier, acc_glacier, runoff_glacier] = display_output_glacier(point_output_path_glacier, yr, mm, dd, date);
end
end
These lines are themselves part of a function. In it, I use the resulting variables (eg: albedo_land, albedo_glacier etc..) to make some calculations and plots. Originally I just ran the functions by writing :
[T20cm_land, albedo_land, smb_land, swe_land, T1m_land, T0_land, abl_land,...
acc_land, runoff_land] = display_output_land(point_output_path_land, yr, mm, dd, date);
[T20cm_glacier, albedo_glacier, smb_glacier, swe_glacier, T1m_glacier, T0_glacier,...
abl_glacier, acc_glacier, runoff_glacier] = display_output_glacier(point_output_path_glacier, yr, mm, dd, date);
But these two functions are independant and take around 30 minutes to run. Is there a way to save all the output variables from these 2 functions outside of the parfor loop, in order to use them later in the script ?
Have a nice day !

采纳的回答

Edric Ellis
Edric Ellis 2020-4-29
Output variables from parfor must be either sliced or reduction variables. So, you could adapt your code like so:
parfor i = 1:2
if i == 1
[T20cm{i}, albedo{i}, . . .] = display_output_land(. . .);
else
[T20cm{i}, albedo{i}, . . .] = display_output_glacier(. . .);
end
end
This converts all the outputs into sliced form.
An alternative given the fact that you really essentially have two completely separate function calls is to use parfeval instead, like this:
% Invoke both function evaluations in parallel.
fut_land = parfeval(@display_output_land, 9, point_output_path_land, yr, mm, dd, date);
fut_glacier = parfeval(@display_output_glacier, 9, point_output_path_glacier, yr, mm, dd, date);
% Collect results - this blocks until the evaluation is complete.
[T20cm_land, albedo_land, . . .] = fetchOutputs(fut_land);
[T20cm_glacier, albedo_glacier, . . .] = fetchOutputs(fut_glacier);
  3 个评论
Edric Ellis
Edric Ellis 2020-4-29
Glad you got things working!
Saving files within parfor is generally not a great idea - and not just from an efficiency standpoint - it might work in a simple case like this providing you can ensure the workers are always writing to different files, but it's easy to trip up if you end up trying to write to the same files from different workers. Also, if you scaled up to a remote cluster, then there's a chance that the client MATLAB might not be able to see the same filesystem as the workers.
V.D-C
V.D-C 2020-4-29
Then I should change my code for your solution, I'm working on a cluster (even though the files are written in a different folder). Thank you for your advices !

请先登录,再进行评论。

更多回答(0 个)

类别

Help CenterFile Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by