Memory issues using parfeval

16 次查看（过去 30 天）
Andre Zeug
Andre Zeug 2020-4-22
回答： Swastik Sarkar 2024-10-16
Hi everyone!
Since years I am using the nice function export_fig to save figures, which I create with my scripts and functions. To speed this up I tried to do this in background using 'parfeval'. Unfortunately, for some reason workers pile up memory consumption which can be monitored e.g. with the ProcessExplorer.
function export_figP(fname, varargin)
% same input as export_fig, but first input must be the filename, last
% input must be the figure handle (could be improved)
tParFeval = tic;
tt = []; fig_1 = [];
set(varargin{end},'Visible','off')
myEvalStr = 'tt = parfeval(@export_fig, 0, fname';
for n=1:nargin-2
myEvalStr = sprintf('%s, ''%s''',myEvalStr, varargin{n});
end
myEvalStr = sprintf('%s, fig_1);',myEvalStr);
n = nargin-1;
eval(sprintf('fig_1 = varargin{%d};',n))
eval(myEvalStr)
afterEach(tt, @(~)evalAfterAll(tt,fig_1,fname,tParFeval), 0);
function evalAfterAll(tt,hh,saveStr,tParFeval)
if ~isempty(tt.Error)
fprintf(2,'!!! Error saving Figure %s !!! \n',saveStr)
else
fprintf(2,'Figure ''%s'' saved within %.3fs (total time %.3fs)\n',saveStr,(datenum(tt.FinishDateTime)-datenum(tt.CreateDateTime))*24*60*60, toc(tParFeval))
close(hh)
tt = [];
clear all
end
end
end
The function can be tested with this lines (requires export_fig)
%% Test export_figP
if ~isfolder(fullfile(pwd, 'figs')); mkdir('figs');end
tic
for n = 1:1000
% create simple figure
fig1 = figure('Color','w','Visible','on');
imagesc(rand(1000, 1000))
colorbar
% save figure in Background
saveStr = sprintf('%04d_TestFig.png',n);
export_figP(sprintf('figs/%s',saveStr),'-q100','-nocrop','-r300',fig1);
fprintf('saving of %s started (%.3f)\n', saveStr, toc)
end
What do I do wrong?
How to clear the workers memory after the function call (except shuting down parallel pool)?
Probably my function 'export_figP' does something substantially wrong.
  1 个评论
Andre Zeug
Andre Zeug 2020-5-1
Is there an option to clear workers memory when no further job is processed?
p = gcp('nocreate')
p =
ProcessPool with properties:
Connected: true
NumWorkers: 16
Cluster: local
AttachedFiles: {}
AutoAddClientPath: true
IdleTimeout: Inf (no automatic shut down)
SpmdEnabled: true
q = p.FevalQueue
q =
FevalQueue with properties:
Number Queued: 0
Number Running: 0
But memory consumtion is still big after several calls of parfeval:

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

Swastik Sarkar
Swastik Sarkar 2024-10-16
Hi @Andre Zeug,
The export_figP and evalAfterAll functions can be enhanced for readability and effective management of figures throughout their lifecycle. Below is the improved implementation:
function export_figP(fname, varargin)
figHandle = varargin{end};
set(figHandle, 'Visible', 'off');
tt = parfeval(@export_fig, 0, fname, varargin{1:end-1}, figHandle);
afterEach(tt, @(~) evalAfterAll(tt, figHandle, fname), 0);
end
function evalAfterAll(tt, figHandle, saveStr)
if ~isempty(tt.Error)
fprintf(2, '!!! Error saving Figure %s !!! \n', saveStr);
else
fprintf('Figure ''%s'' saved successfully.\n', saveStr);
end
% Close the figure and clear the task
close(figHandle);
delete(tt);
end
Clearing the output data of the parallel.FevalFuture object can further enhance performance. More information can be found in the following resource:
https://www.mathworks.com/matlabcentral/answers/424473-parfeval-memory-consumption-piling-up-clear-output-data#answer_342050
Hope this helps.

类别

在 Help Center 和 File Exchange 中查找有关 Text Data Preparation 的更多信息

标签

产品


版本

R2020a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by