Use Parfor to split a 4D matrix into several 3D matrices

1 次查看（过去 30 天）
N/A
N/A 2018-9-4
编辑： Stephen23 2018-9-4
Hey,
so I to change my current code:
for i=1:d(4)
save_file_name=sprintf('layer_%i.mat',i); %name of the save file
variable_name=sprintf('layer_%i',i); %name of the variable
reduced_dimension=squeeze(reducedData(:,:,:,i)); %reduces the size of the matrix down to 3 since
eval(['layer_' num2str(i) '=reduced_dimension;']);%renames the variable
save(save_file_name,variable_name);%saves the files
eval(['clear ', 'layer_' num2str(i) ])
end
This code is supposed to split a giant 4D Matrix into several smaller 3D matrices while automatically renaming and saving them separatly. I am aware that using eval is considered to be bad, but so far I have not found any other solution.
I want to speed up the whole process by using a parfor loop, but apparently "eval" and "save" are not allowed in this kind of loop. Has anyone an idea how to solve this kind of problem ?
  2 个评论
OCDER
OCDER 2018-9-4
What you're trying to do seems to be going in the wrong direction in terms of code efficiency... Seems like you already have the code to split a 4D to 3D, and you're already aware that using eval to rename variables is a bad practice. Note that using parfor won't help much because you are limited by the hard drive write speed. It'll be better if you tell us:
  • What is the real task you'd like to accomplish AFTER you split the 4D matrix?
  • Why do you need to split this 4D matrix?
Stephen23
Stephen23 2018-9-4
编辑：Stephen23 2018-9-4
"Use Parfor to split a 4D matrix into several 3D matrices"
You can already access the data in the 4D array very efficiently using indexing (and indexing is very efficient). In contrast, what you are trying to do with eval will only force you into writing slow, complex, buggy code. For example, by using eval you remove any ability of the MATLAB JIT engine to speed up your code, so you automatically force your code into being pointlessly slow, all of the time:
https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval
If you used simple indexing then MATLAB would speed up your code automatically.
"I want to speed up the whole process by using a parfor loop, but apparently "eval" and "save" are not allowed in this kind of loop"
If you are planning on writing fast and efficient code (which is usually what beginners intend when they start talking about parfor) then your approach is in entirely the wrong direction. If you really want to write efficient code:
  • do NOT make unnecessary duplicates of data in memory.
  • do NOT use eval or anything else to magically access variable names.
  • follow the guidelines in the MATLAB documentation:
https://www.mathworks.com/help/matlab/matlab_prog/techniques-for-improving-performance.html

请先登录，再进行评论。

请先登录，再回答此问题。

回答（2 个）

Titus Edelhofer
Titus Edelhofer 2018-9-4
Hi,
I agree with OCDER: parfor will only do huge memory movements from the Client to the workers but I doubt you will get much speedup.
If the "save" is indeed too slow, you might experiment with saving without compression: use "-v6" as flag when saving. The files are then probably much bigger but often saving is faster.
And by the way: if you have the file name with the layer number, why bothering what the variable name is? Why don't simply use "layer" for all of them?
Titus
Stephen23
Stephen23 2018-9-4
编辑：Stephen23 2018-9-4
Firstly it is important to note that processing a sequence of .mat files is much simpler when the variable names inside each file are exactly the same. This applies to both reading and writing them. Then you could trivially do this:
for slice = 1:d(4)
file = sprintf('layer_%d.mat',slice);
data = squeeze(reducedData(:,:,:,slice));
save(file,'data','slice')
end
Note how I wrote simpler, much more efficient code than what you have in your question, and that I saved the slice value too, so you know exactly which slice it is when you load that data:
S = load(...)
S.slice
S.data
Simpler to write, simpler to read, much more efficient: why waste your time fighting eval ?
If you really want to have lots of numbered variables (which are a sign of bad data design, and bad code), then you could split the data into the fields of a structure, and save that structure using the '-struct' option. But of course that would be slow, complex, and not recommended.

类别

在 Help Center 和 File Exchange 中查找有关 Loops and Conditional Statements 的更多信息

标签

产品


版本

R2017b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by