Desired object produced but still receive "out of memory error"

1 次查看(过去 30 天)
I encountered something quite odd and was hoping someone could provide some insight.
I have a very large variable ("W") that I need to run through some logical indexing in the form of a for loop. The end result should be a 110000 X 5000 X 401 3D matrix, but 220.5 billion numbers is a bit much for memory to handle, even if I make it a sparse array. You can see I've broken "W" up into 11 parts to see if that works.
This code ran for me and produced the object "b_1", but I still got an out of memory error. The out of memory error suggests to me that the object should have never been produced. I'm not sure that I trust the produced object "b_1" because of the out of memory error. Am I being too paranoid, or would it be wise to look for a different solution?
T = 5000
W = zeros(110000, 5000);
for trial = 1:T
X2ri = X;
X2ri(10001:20000, 46:50) = randi([1,5], 10000, 5);
X3ri = X2ri;
X3ri(20001:30000, 41:50) = randi([1,5], 10000, 10);
X4ri = X3ri;
X4ri(30001:40000, 36:50) = randi([1,5], 10000, 15);
X5ri = X4ri;
X5ri(40001:50000, 31:50) = randi([1,5], 10000, 20);
X6ri = X5ri;
X6ri(50001:60000, 26:50) = randi([1,5], 10000, 25);
X7ri = X6ri;
X7ri(60001:70000, 21:50) = randi([1,5], 10000, 30);
X8ri = X7ri;
X8ri(70001:80000, 16:50) = randi([1,5], 10000, 35);
X9ri = X8ri;
X9ri(80001:90000, 11:50) = randi([1,5], 10000, 40);
X10ri = X9ri;
X10ri(90001:100000, 6:50) = randi([1,5], 10000, 45);
X11ri = X10ri;
X11ri(100001:110000, 1:50) = randi([1,5], 10000, 50);
XFri = X11ri;
W(:,trial) = var(XFri,0,2);
end
W_1 = W(1:10000,:);
W_2 = W(10001:20000,:);
W_3 = W(20001:30000,:);
W_4 = W(30001:40000,:);
W_5 = W(40001:50000,:);
W_6 = W(50001:60000,:);
W_7 = W(60001:70000,:);
W_8 = W(70001:80000,:);
W_9 = W(80001:90000,:);
W_10 = W(90001:100000,:);
W_11 = W(100001:110000,:);
varinc = (0:0.01:4);
b_1 = ndSparse.build([10000,5000,401]);
for i = 1:length(varinc)
b_1(:,:,i) = W_1 >= varinc(i);
end

采纳的回答

Walter Roberson
Walter Roberson 2020-9-26
you are using https://www.mathworks.com/matlabcentral/fileexchange/29832-n-dimensional-sparse-arrays
You need to use nzmax to preallocate the object. Otherwise every time you assign into the object, matlab needs to create a copy of the array with one more value slot and copy the old one over.
  4 个评论
Walter Roberson
Walter Roberson 2020-9-26
Yes, I figure that at some point when it went to make a copy to be able to expand the array, it ran out of memory, and the result you got was filled in to that point only.
I always recommend using spalloc or providing nzmax when creating an array using sparse. The overhead for expanding a sparse array is not fun.
Also be careful about
A = B operator C
for sparse B and C. MATLAB will strip out any unallocated locations in the result.
May_the_degrees_of_freedom_be_with_you
Hi Walter,
Thank you again for your insight! This has given me a lot to think about, but it will make my future code much better.

请先登录,再进行评论。

更多回答(0 个)

类别

Help CenterFile Exchange 中查找有关 Matrix Indexing 的更多信息

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by