Memory efficient alternative to sprandsym?

Question

Philipp 2023-6-23

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1987314-memory-efficient-alternative-to-sprandsym

评论： Philipp 2023-6-29

Hello,

I try to create a sparse symmetric random matrix according to some pattern using the command sprandsym. It appears to me, however, that sprandsym consumes a lot of memory during the construction of the matrix even if the final matrix needs much less memory. Obviously, I like to avoid this.

Here is a minimal example where I use two lists (each has 150 elements) to create two matrices, which at the end, by cutting away elements above a certain threshold, determine the pattern of the random matrix:

listA = readmatrix('data/listA.csv');
listB = readmatrix('data/listB.csv');
A = listA(:) - listA(:)'; 
B = listB(:) - listB(:)'; 
Pattern_A = sparse(A >= -threshold & A <= threshold & A ~= 0);
Pattern_B = sparse(B >= -threshold & B <= threshold & B ~= 0);
Pattern = kron(Pattern_A,Pattern_B);

Now, everything works fine until here and the code consumes a bit less than 3 GB. But then, the next line of the code is

RM = sprandsym(Pattern);

During the computation of sprandsym the used memory grows up to 14 GB, but after execution of sprandsym the used memory is around 6.7 GB. This seems to indicate that only the execution of sprandsym seems to consume almost 10 GB extra memory, but I do not really understand why. Is there any explanation? And, ideally, is there any way to make it better?

If it helps, it turns out that the matrix Pattern is not extremely sparse (density of elements around 0.3). Maybe this explains it? I could reduce the density, say to 0.1, but I really like to avoid the extra use of memory during the execution of sprandsym.

Any help is greatly appreciated. Have a nice weekend!

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Jan 2023-6-25

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1987314-memory-efficient-alternative-to-sprandsym#answer_1262048

在 MATLAB Online 中打开

It would be useful to create the input by some commands in the example, such that the readers can run the code without your files.

Does this simplification change the memory consumption:

A = sparse(listA(:) - listA(:)'); 
B = sparse(listB(:) - listB(:)'); 
Pattern_A = (A >= -threshold & A <= threshold & A);
Pattern_B = (B >= -threshold & B <= threshold & B);
Pattern   = kron(Pattern_A, Pattern_B);

3 个评论
显示 1更早的评论隐藏 1更早的评论

Jan 2023-6-27

在 MATLAB Online 中打开

Creating the input by code means to replace

listA = readmatrix('data/listA.csv');

by code, which constructs meaningful testdata, e.g.:

listA = randi([0, 1], 1, 150)

It would be useful to answer questions for clarifications instead of ignoring them: Does this simplification change the memory consumption? Yes or no?

How do you measure the memory consumption?

Philipp 2023-6-29

在 MATLAB Online 中打开

I see. Yes, one can put

listA=rand(150,1);

and set

threshold = 0.3;

The same problem appears.

And no, your initial solution didn't change anything. Also, A and B are not sparse at all.

请先登录，再进行评论。

Answer 2

Philipp 2023-6-27

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1987314-memory-efficient-alternative-to-sprandsym#answer_1263248

I have checked this further and it appears to me that part of the problem is that the matrix is not sparse enough. Indeed, I have just created a full random matrix (without any sparsity) of the same dimension and this consumed significantly less memory (around 7 GB) than sprandsym consumed for my not too sparse matrix.

If I increase the sparsity, say to a density of elements around 0.1, sprandsym becomes more efficient and uses only about 4 GB of memory. Still, storing the final sparse matrix in the memory consumes only around 2 GB of memory.

In this sense I am still puzzled how sprandsym works exactly and why it consumes significantly more memory than the final sparse matrix requires for storage?!...