Desired object produced but still receive "out of memory error"

Question

May_the_degrees_of_freedom_be_with_you 2020-9-25

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/600124-desired-object-produced-but-still-receive-out-of-memory-error

评论： May_the_degrees_of_freedom_be_with_you 2020-9-26

I encountered something quite odd and was hoping someone could provide some insight.

I have a very large variable ("W") that I need to run through some logical indexing in the form of a for loop. The end result should be a 110000 X 5000 X 401 3D matrix, but 220.5 billion numbers is a bit much for memory to handle, even if I make it a sparse array. You can see I've broken "W" up into 11 parts to see if that works.

This code ran for me and produced the object "b_1", but I still got an out of memory error. The out of memory error suggests to me that the object should have never been produced. I'm not sure that I trust the produced object "b_1" because of the out of memory error. Am I being too paranoid, or would it be wise to look for a different solution?

T = 5000
W = zeros(110000, 5000);
for trial = 1:T
    X2ri = X;
    X2ri(10001:20000, 46:50) = randi([1,5], 10000, 5);
    X3ri = X2ri;
    X3ri(20001:30000, 41:50) = randi([1,5], 10000, 10);
    X4ri = X3ri;
    X4ri(30001:40000, 36:50) = randi([1,5], 10000, 15);
    X5ri = X4ri;
    X5ri(40001:50000, 31:50) = randi([1,5], 10000, 20);
    X6ri = X5ri;
    X6ri(50001:60000, 26:50) = randi([1,5], 10000, 25);
    X7ri = X6ri;
    X7ri(60001:70000, 21:50) = randi([1,5], 10000, 30);
    X8ri = X7ri;
    X8ri(70001:80000, 16:50) = randi([1,5], 10000, 35);
    X9ri = X8ri;
    X9ri(80001:90000, 11:50) = randi([1,5], 10000, 40);
    X10ri = X9ri;
    X10ri(90001:100000, 6:50) = randi([1,5], 10000, 45);
    X11ri = X10ri;
    X11ri(100001:110000, 1:50) = randi([1,5], 10000, 50);
    XFri = X11ri;
    W(:,trial) = var(XFri,0,2);
end
W_1 = W(1:10000,:);
W_2 = W(10001:20000,:);
W_3 = W(20001:30000,:);
W_4 = W(30001:40000,:);
W_5 = W(40001:50000,:);
W_6 = W(50001:60000,:);
W_7 = W(60001:70000,:);
W_8 = W(70001:80000,:);
W_9 = W(80001:90000,:);
W_10 = W(90001:100000,:);
W_11 = W(100001:110000,:);
varinc = (0:0.01:4);
b_1 = ndSparse.build([10000,5000,401]);
for i = 1:length(varinc)
    b_1(:,:,i) = W_1 >= varinc(i);
end

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2020-9-26

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/600124-desired-object-produced-but-still-receive-out-of-memory-error#answer_500692

you are using https://www.mathworks.com/matlabcentral/fileexchange/29832-n-dimensional-sparse-arrays

You need to use nzmax to preallocate the object. Otherwise every time you assign into the object, matlab needs to create a copy of the array with one more value slot and copy the old one over.

4 个评论
显示 2更早的评论隐藏 2更早的评论

May_the_degrees_of_freedom_be_with_you 2020-9-26

Hi Walter,

Thank you for the insight! I should look into nzmax--that should get this to run quicker and without memory errors (hopefully). I thought I was preallocating with ndSparse.build, but it does appear that the documentation recommends nzmax. I wonder if that's something specific to the ndSparse "add-in"...

For anyone following or who might encounter a similar issue: I tried running this again after clearing my workspace of variables I wasn't currently using in the above code. After doing that, the abbreviated 10000X5000X401 3D array ran, and it came out as a "logical" object type and with no memory error. The object produced before (when an object was produced but I still got a memory error) was still of the ndSparse type. I'm not 100% sure if this is the right conclusion, but based on the final object with no memory error being of the "logical" type, I'm concluding that the memory error indicated that the ndSparse object that I originally produced (the one that came with a memory error) wasn't completely filled in.

May_the_degrees_of_freedom_be_with_you 2020-9-26

在 MATLAB Online 中打开

Hi Walter,

I was wondering if you might provide some insight as to how to use the nzmax subcommand. Even prior to today, I've spent a deal of time reading over the materials for ndSparse and sparse/spalloc, but I can't decipher what to do about using nzmax for preallocation.

For example, if I write the following to preallocate space for 300 nonzero values

temp = ndSparse.build([500,100],300)

the returned array is of the correct dimensions, but when I inspect it using full(temp) the above code just placed a "300" in the position temp(500,100) (or the bottom cell of the last column). It doesn't seem to be doing what it should, but that code should follow the given formula from the documentation: S=ndSparse.build(Coordinates,Values,[m,n,p,...],nzmax)

I'm also unsure of what to use for any nzmax value, as my purpose for this specific analysis is that I want to determine how many nonzero values the final array contains after logical indexing.

Thank you again for your insight.

Walter Roberson 2020-9-26

在 MATLAB Online 中打开

Yes, I figure that at some point when it went to make a copy to be able to expand the array, it ran out of memory, and the result you got was filled in to that point only.

I always recommend using spalloc or providing nzmax when creating an array using sparse. The overhead for expanding a sparse array is not fun.

Also be careful about

A = B operator C

for sparse B and C. MATLAB will strip out any unallocated locations in the result.

May_the_degrees_of_freedom_be_with_you 2020-9-26

Hi Walter,

Thank you again for your insight! This has given me a lot to think about, but it will make my future code much better.

请先登录，再进行评论。

Desired object produced but still receive "out of memory error"

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

4 个评论
显示 2更早的评论隐藏 2更早的评论

更多回答（0 个）

另请参阅

类别

标签

Community Treasure Hunt

Desired object produced but still receive "out of memory error"

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

4 个评论 显示 2更早的评论隐藏 2更早的评论

更多回答（0 个）

另请参阅

类别

标签

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

4 个评论
显示 2更早的评论隐藏 2更早的评论