inmem - list files loaded inside parfor

3 次查看(过去 30 天)
It looks like inmem does not list functions loaded inside parfor loops. At least that is what I'm experiencing in R2014b and I have not found this mentioned in the docs. Is this expected behaviour?
I understand it's the workers who execute the file, but the master process could be made aware of which files are loaded by the workers ... it already communicates with all workers back and forth, and waits for all workers to finish anyway.
This would be extremely useful for reproducibility in research. I could save/zip all files that were loaded for the execution chain and also save the inputs as .mat and the last history entry, and then be able to completely reproduce the same results later. As it is, not all files are saved and that's a shame.
I know I could use matlab.codetools.requiredFilesAndProducts but that saves all dependencies (not just those needed for the particular set of inputs I used), which in my case are more than 100 MB worth of files, as opposed to 100 KB with inmem.
Specifically:
parfor k=1:100
external_file(k);
end
[files,mexs] = inmem('-completenames');
The above does not list 'external_file.m'. Replacing parfor with for will list it, but then my code is slow ...
Best, Alex.

回答(1 个)

Walter Roberson
Walter Roberson 2015-5-25
inmem is documented specifically as being the ones that are currently loaded, not a history of what has ever been loaded during the session. If a function has been cleared, it is no longer loaded. The workers logically unload after a parfor, so the loaded file is logically gone.
If workers do not unload after parfor, then you could, after a parfor, parfor "enough" to use all of the worker entries:
%maybe 50 will be enough to hit each worker at least once. Maybe not.
was_inmem = cell(50,1);
parfor K = 1 : 50
was_inmem = inmem('-completenames');
end
was_inmem = unique({was_inmem{:}});
My prediction is that it won't show anything.
Safer way: incorporate the above strategy right into your original parfor
was_inmem = cell(100,1);
parfor k=1:100
external_file(k);
was_inmem{k} = inmem('-completenames');
end
was_inmem = unique({was_inmem{:}});
  4 个评论
Edric Ellis
Edric Ellis 2015-6-1
An alternative to pctRunOnAll in >= R2013b is:
f = parfevalOnAll(@inmem, 1)
which runs only the workers and returns you a future from which you can extract the individual results from each worker.
Walter Roberson
Walter Roberson 2015-6-1
Some day when I win the lottery, I'm gonna buy me a copy of PCT to play with...

请先登录,再进行评论。

类别

Help CenterFile Exchange 中查找有关 Loops and Conditional Statements 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by