Efficiently moving values in a 3-dimensional array

7 次查看（过去 30 天）
MATLAB_learner
MATLAB_learner 2023-7-24
编辑： Bruno Luong 2023-7-26
I am looking for a clever way to use logical indexing to swiftly move large numbers of values around in a large array.
I have a large 3-dimensional array (called 'my_array'). Each row is individual, columns are information about each individual, and slices are different populations of individuals.
I have another array called 'changes_to_make' that describes how I want to modify 'my_array' array by moving rows around. Specifically, there are a large number of rows in 'my_array' that I want to move within their specific slice. For instance, I might want to move the 55th individual of the 34th population (i.e., all values in row 55 of slice 34) to the 22nd row of the same population. Each row of the 'changes_to_make' array contains the information needed for a single change: column 1 is the starting row, column 2 is the destination row, and column 3 is the slice of 'my_array' in which I want to make that specific change. There are thousands of changes to make.
Once a row has been moved somewhere else, the original row is converted to 'NaN'.
None of the changes are overlapping: they all involve independent combinations of rows, and in no cases is a row moved twice. Therefore, all changes can be implemented in one go, using logical indexing, rather than needing a for-loop.
My problem is that this is currently taking a long time (several seconds). Ultimately, this will be part of a simulation iterated thousands of times, so taking several seconds to complete the changes means the total simulation is very slow.
Is there a way to improve the efficiency of the process?
Many thanks if you can help.
To set up a reproducible example:
n_rows = 400;
n_columns = 100;
n_populations = 10000;
my_array = rand(n_rows,n_columns,n_populations); % generate an example 3-dimensional array
% Generate an example 'changes_to_make' array:
changes_to_make = nan(n_rows/2,3,n_populations);
for slice = 1:n_populations
changes_to_make(:,1,slice) = (1:2:n_rows)';
a = (2:2:n_rows)'; changes_to_make(:,2,slice) = a(randperm(length(a)));
changes_to_make(:,3,slice) = slice;
end
changes_to_make = reshape(permute(changes_to_make, [2 1 3]), size(changes_to_make, 2), [])';
My current solution is the following, which is very slow:
tic
for column_to_edit = 1:n_columns
my_array(sub2ind(size(my_array),changes_to_make(:,2),ones(length(changes_to_make),1).*column_to_edit,changes_to_make(:,3))) = my_array(sub2ind(size(my_array),changes_to_make(:,1),ones(length(changes_to_make),1).*column_to_edit,changes_to_make(:,3)));
my_array(sub2ind(size(my_array),changes_to_make(:,1),ones(length(changes_to_make),1).*column_to_edit,changes_to_make(:,3))) = NaN;
end
toc
Elapsed time is 7.053371 seconds.
  1 个评论
Matt J
Matt J 2023-7-24
编辑：Matt J 2023-7-24
my_array = rand(n_rows,n_columns,n_populations)
It will be much more efficient if you use the data ordering
my_array = rand(n_columns,n_rows, n_populations)

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Matt J
Matt J 2023-7-24
编辑：Matt J 2023-7-24
Ran in:
It will be much more efficient if you use the data ordering my_array = rand(n_columns,n_rows, n_populations)
I will assume you've taken this advice. Then, it will simply be,
n_rows = 400;
n_columns = 100;
n_populations = 10000;
my_array = rand(n_columns,n_rows,n_populations,'single'); % generate an example 3-dimensional array
% Generate an example 'changes_to_make' array:
changes_to_make = nan(n_rows/2,3,n_populations);
for slice = 1:n_populations
changes_to_make(:,1,slice) = (1:2:n_rows)';
a = (2:2:n_rows)'; changes_to_make(:,2,slice) = a(randperm(length(a)));
changes_to_make(:,3,slice) = slice;
end
changes_to_make = reshape(permute(changes_to_make, [2 1 3]), size(changes_to_make, 2), [])';
tic;
start=sub2ind( [n_rows,n_populations],changes_to_make(:,1),changes_to_make(:,3) );
dest=sub2ind( [n_rows,n_populations],changes_to_make(:,2),changes_to_make(:,3) );
result=my_array;
result(:,dest)=my_array(:,start);
result(:,start)=nan;
toc
Elapsed time is 1.518013 seconds.
  2 个评论
Matt J
Matt J 2023-7-24
编辑：Matt J 2023-7-24
However, I don't think orders of magnitude speed-up are possible. I think you are limited just by the sheer size of your data. Notice that I've used single instead of double floats to help with the speed-up. You should do that if you can afford the loss of precision.

请先登录，再进行评论。

更多回答（1 个）

Bruno Luong
Bruno Luong 2023-7-25
编辑：Bruno Luong 2023-7-26
Ran in:
I'm with Matt on permuting your data. I recommend not moving data around but rebuild a new array, since I suspect some deep copy of entire data occurs twice.
n_rows = 400;
n_columns = 100;
n_populations = 10000;
my_array = rand(n_columns,n_rows,n_populations); % generate an example 3-dimensional array
% Generate an example 'changes_to_make' array:
changes_to_make = nan(n_rows/2,3,n_populations);
for slice = 1:n_populations
changes_to_make(:,1,slice) = (1:2:n_rows)';
a = (2:2:n_rows)'; changes_to_make(:,2,slice) = a(randperm(length(a)));
changes_to_make(:,3,slice) = slice;
end
changes_to_make = reshape(permute(changes_to_make, [2 1 3]), size(changes_to_make, 2), [])';
tic;
start=sub2ind( [n_rows,n_populations],changes_to_make(:,1),changes_to_make(:,3) );
dest=sub2ind( [n_rows,n_populations],changes_to_make(:,2),changes_to_make(:,3) );
result=my_array;
result(:,dest)=my_array(:,start);
result(:,start)=nan;
toc
Elapsed time is 2.789713 seconds.
tic
tmp = n_rows*(changes_to_make(:,3)-1);
start = tmp + changes_to_make(:,1);
dest = tmp + changes_to_make(:,2);
col = 1:n_rows*n_populations;
col(dest) = start;
cnan=dest(1);
col(start) = cnan;
my_array(:,cnan) = NaN; % MATLAB seems to do inplace replacement here, R2023a
result2=my_array(:,col);
result2=reshape(result2,size(my_array));
toc
Elapsed time is 1.393631 seconds.
% Check correctness
d=(result-result2);
d(isnan(d))=0;
all(d(:)==0)
ans = logical
1
  3 个评论
显示 1更早的评论
Matt J
Matt J 2023-7-26
It's interesting that this gives a faster time, though I don't really understand why, unless cnan somehow contains fewer columns than start.
Bruno Luong
Bruno Luong 2023-7-26
编辑：Bruno Luong 2023-7-26
I suspect there are two full deep data copying in your code, when these two statement are performed
result(:,dest)=my_array(:,start);
result(:,start)=nan;
cnan is a scalar, so
my_array(:,cnan) = NaN; % MATLAB seems to do inplace replacement here, R2023a
replaces a single column of my_array; and MATLAB does inplace replacement and not deep copy.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Logical 的更多信息

标签

产品


版本

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by