Parsing files and computing in parfor - GA

2 次查看（过去 30 天）
Dominik Maly
Dominik Maly 2024-3-11
回答： Dominik Maly 2024-3-12
Hey, I've been working on some optimization problems within GA. There are some problems with files when i try use my "work remote" cluster (MJS Cluster).
I've got 2 main files:
'Fused_VP_40u.mat" - 1.26 GB - 1x1688 cell, where every cell contains couple thousands rows and always 50 columns,
''Fused_VP_40u_20t.mat" - 652 MB - 1x1688 cell, where every cell contains couple thousands rows and always 50 columns.
My task is to optimize these 50 columns to find the best combination (So I'm trying to lower the number of columns and also increase the accuracy - finding the best solution)
When I'm using it on my PC, with parfors inside it's working. Not so fast but working. Now i've got remote cluster with 32 workers which I want to use in my computations. I've got problems with parsing these files to workers and my script ends.
Here is the main code snippet:
load('Fused_VPs_40u.mat')
load('Fused_VPs_40u_20t.mat')
load('Features_chosen1.mat')
load('Features_chosen_new.mat','Feats_chosen')
%%
Feats_num = 50;
x_start = ones(1,50);
ub = ones(1,Feats_num)
lb = zeros(1,Feats_num);
myFitnessFunction = @(XX) Fit_GMM(XX, VP_merged_u_fused, VP_merged_t_fused);
options = optimoptions("ga","PlotFcn","gaplotbestf",UseParallel=false,Display="iter",PopulationType="bitstring",InitialPopulationMatrix=x_start,PopulationSize=1,MaxStallGenerations=2000,MaxGenerations=5000)
%,MutationFcn="mutationadaptfeasible",CrossoverFraction=0.7,MutationFcn="mutationadaptfeasible",SelectionFcn="selectionroulette",CreationFcn="gacreationlinearfeasible")
% Call GA with the wrapped fitness function
[Best_feats, ff,exitflag,output,population,scores] = ga(myFitnessFunction, Feats_num, [], [], [], [], lb, ub, [], [], options);
% Calculate the accuracy
Acc = 1 - ff;
function errorK = Fit_GMM(XX, VP_merged_t_fused, VP_merged_u_fused,Feats_chosen)
classes = 1:1688;
for g = 1:size(VP_merged_u_fused, 2)
VP_test{g} = VP_merged_t_fused{g} .* XX;
VP_ucz{g} = VP_merged_u_fused{g} .* XX;
GMMs{g} = GMM_VP(VP_ucz{g});
end
clearvars VP_ucz
xxx = size(GMMs, 2);
nrcx = zeros(1, xxx);
for z = 1 : 1688
parfor i = 1 : size(GMMs, 2)
[~, nllx(i)] = posterior(GMMs{i}, VP_test{z});
end
nllx = nllx / size(VP_test{z}, 1);
[~, nrcx(z)] = min(nllx);
end
Acc = sum(nrcx == classes) / length(classes);
errorK = 1 - Acc;
XX_n = [XX,Acc];
Feats_chosen = [Feats_chosen; XX_n];
save Features_chosen_new Feats_chosen
end
Here is problem i've faced:
Warning: Could not find appropriate function on path loading function handle C:\Users\XXX\AppData\Local\Temp\Editor_rlrqf\LiveEditorEvaluationHelperE1805384047.m>makeF%1/F%0
> In parallel.internal.pool.optionallyDeserialize (line 7)
In parallel.internal.parfor.cppRemoteParallelFunction (line 25)
Error using GA_fusion>Fit_GMM
The source code (D:\XXXXX\XXXXX\GA_fusion.m) for the parfor-loop that is trying to execute on the worker could not be found.
Error in GA_fusion>@(XX)Fit_GMM(XX,VP_merged_u_fused,VP_merged_t_fused,Feats_chosen) (line 24)
myFitnessFunction = @(XX) Fit_GMM(XX, VP_merged_u_fused, VP_merged_t_fused,Feats_chosen);
Error in createAnonymousFcn>@(x)fcn(x,FcnArgs{:}) (line 11)
fcn_handle = @(x) fcn(x,FcnArgs{:});
Error in makeState (line 58)
firstMemberScore = FitnessFcn(state.Population(initScoreProvided+1,:));
Error in gaunc (line 47)
state = makeState(GenomeLength,FitnessFcn,Iterate,output.problemtype,options);
Error in ga (line 417)
[x,fval,exitFlag,output,population,scores] = gaunc(FitnessFcn,nvars, ...
Error in GA_fusion (line 28)
[Best_feats, ff,exitflag,output,population,scores] = ga(myFitnessFunction, Feats_num, [], [], [], [],
lb, ub, [], [], options);
Caused by:
Undefined function handle.
Error using parallel.internal.parfor/cppRemoteParallelFunction
Worker unable to find file.
Undefined function handle.
Failure in initial user-supplied fitness function evaluation. GA cannot continue.
I've tried using "matfile" instead of load - didn't work.
Tried using AddAttachedFiles - also didn't work (connection is taking very long and lost over some time).
Maybe I should load these variables in other way?

请先登录，再回答此问题。

回答（2 个）

Edric Ellis
Edric Ellis 2024-3-11
When working with a remote cluster parpool, usually Parallel Computing Toolbox is able to work out automatically which files need to be transferred to the cluster to make things work. However, this isn't always the case. A couple of things to try:
  1. use addAttachedFiles to add the file containing Fit_GMM to your parallel pool
  2. If that doesn't help - try running the entire script from the command-line. It looks like you might be using the editor to run sections of your script rather than the whole thing. This can sometimes interfere with the auto-attaching of files.
Dominik Maly
Dominik Maly 2024-3-12
So today I solved this problem.
To make things work i've replaced:
load('Fused_VPs_40u.mat')
load('Fused_VPs_40u_20t.mat')
with:
var1 = matfile('Fused_VPs_40u.mat')
var2 = matfile('Fused_VPs_40u_20t.mat')
var_to_ga1 = var1.variable_name_we_want_to_process_from_matfile;
var_to_ga2 = var2.variable_name_we_want_to_process_from_matfile;
After this change everything works and runs smoothly.

类别

在 Help Center 和 File Exchange 中查找有关 Parallel Computing Fundamentals 的更多信息

标签

产品


版本

R2023b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by