Surrogateopt does not execute objective function in parallel and still uses 120GB of RAM

6 次查看（过去 30 天）
Florian
Florian 2023-7-21
回答： Ashutosh Thakur 2023-12-19
Hi all,
I am using surrogateopt to optimize an expensive objective function. One evaluation takes approx. 6 minutes.
I run it on a Win 10 PC with 28 cores and 128GB RAM and Matlab 2023a with the Global Optimization Toolbox and the Parallel Computing Toolbox.
Each evaluation of the objective function calls a legacy fortran executable twice. The function value is an average of the two fortran outputs. (see pseudo code below)
The fortran executable is 32 bit and needs ca. 1.5GB of RAM. By my math, the PC should be good for 27 parallel executions of that fortran file.
Each time surrogateopt starts, it does indeed start Poolsize number of parallel fortrans, however, it reverts to serial execution afterwards. At the same time, Matlab uses all available 128GB of RAM. See screenshot attached.
Does anyone have an idea of how to make surrogateopt work "more parallel?" Does surrogateopt have such a high overhead in terms of RAM?
As a sidenote: if I use patternsearch in parallel, I am able to execute 10 (2*length(x)) instances consuming approx. 15 GB of RAM. However, many cores sit idle and I hoped surrogate could help.
Thank you very much!
Florian
lb=[.5 0 0 0 ];
x0=[.9 .5 .2 .25];
ub=[ 1 2 .5 10 ];
opts = optimoptions('surrogateopt',UseParallel=true,UseVectorized=false,Display='iter',PlotFcn='surrogateoptplot',...
InitialPoints=x0,MaxFunctionEvaluations=1e9,MaxTime=3*24*60^2,ObjectiveLimit=1e-4);
fixed_para = [1 2]; %some fixed parameter values
xopt = patternsearch(@(x) objective(x),lb,ub,opts);
function val = objective(x)
for i = 1:2
paraset = [fixed_para(i) , x]; %full set of parameters
%create unique subfolder for each worker
wrkr = getCurrentTask();
folder = [num2str(wrkr.ID),"_",num2str(i)];
if ~isfolder(folder)
copyfile('PricingRevisedDown.exe',folder)
end
%write input file for executable
fid = fopen([folder,'\input.txt'], 'wt');
fprintf(fid,'%s', paraset);
fclose(fid);
%execute the fortran code using input.txt in folder and wait for termination
system(['pushd ',folder,' && START /high /wait PricingRevisedDown.exe && popd']);
%read in results
valtemp(i) = load([folder,'\fortranout.txt']);
end
val = 2.\(valtemp(1) + valtemp(2));
end

请先登录，再回答此问题。

回答（1 个）

Ashutosh Thakur
Ashutosh Thakur 2023-12-19
Hi Florian,
I can understand that you want "surrogateopt" to use less RAM and produce better performance. This may be due to limitations of optimization algorithm or a parallel pool configuration. You can follow these suggestions to improve the performance:
  • You can explicitly create a parallel pool with desired number of workers using the "parpool" function in MATLAB. Try to keep number of workers as maximum.
  • Try to monitor the process usage during execution of the program. Also MATLAB profiler can be used to find the bottlenecks in the program.
  • Also try using different optimization algorithms provided by the Global Optimization Toolbox as some algorithms might be better for your usecase.
  • Try to use different functions such as "parfor", "parfeval" from the Parallel Computing Toolbox to have a better performance.
Please refer to the following documentation links to have a better understanding of the above suggestions:
I hope this helps you in improving the performance.

类别

在 Help Center 和 File Exchange 中查找有关 Programming 的更多信息

标签

产品


版本

R2023a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by