Memory usage in sparse matrix

Question

tanmoy 2014-4-18

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/126295-memory-usage-in-sparse-matrix

编辑： James Tursa 2014-4-21

Hi, Sparse matrices store only the nonzero elements and their position indices. Using sparse matrices can significantly reduce the amount of memory required for data storage. But surprisingly I noticed that my computer runs out of memory when I try something like >>sparse(zeros(80000)) or a matrix having more elements.In zeros(), as there is no non-zero element, the sparse matrix should not have used any memory in this case at all! I would really appreciate if anyone can elucidate this issue... thanks.

2 个评论
显示无隐藏无

Matt J 2014-4-19

编辑：Matt J 2014-4-19

在 MATLAB Online 中打开

as there is no non-zero element, the sparse matrix should not have used any memory in this case at all!

This wasn't your main problem, but it should be pointed out that sparse matrices can in fact consume a great deal of memory, even when containing only zeros. E.g., the following matrix which consumes over 200 MB

>> A=sparse(1,3e7); whos A
    Name      Size                      Bytes  Class     Attributes
    A         1x30000000            240000024  double    sparse

James Tursa 2014-4-21

编辑：James Tursa 2014-4-21

在 MATLAB Online 中打开

@tanmoy: The minimum data storage requirement formula for a double m x n sparse matrix with nnz non-zero elements, including the index data, is as follows on a 32-bit system:

bytes = max(nnz,1) * (4 + 8) + (n+1)*4

Which breaks down as follows:

nnz * 4 = Storing the row index of the non-zero elements
nnz * 8 = Storing the non-zero double element values themselves
(n+1)*4 = Storing the cumulative number of non-zero elements thru column

So you see that even if there are no non-zero elements, the cumulative column index data still gets stored (hence the large memory for Matt J's example). As a user exercise, try sparse(3e7,1) instead.

For 64-bit systems using 8-byte integers for the indexing you can replace the 4's above with 8's.

This is just the minimum requirements. A sparse matrix can have excess memory allocated beyond the minimum if desired.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Ken Atwell 2014-4-19

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/126295-memory-usage-in-sparse-matrix#answer_133854

在 MATLAB Online 中打开

Try just:

>> sparse(80000,80000)

to avoid the temporary allocation of a 80000^2 non-sparse zeros array.

2 个评论
显示无隐藏无

tanmoy 2014-4-19

Thank you for clarifying my queries.The answer is really helpful and it seems to serve my purpose. However, I have one more confusion regarding this.

As I understand, a sparse() matrix in matlab only stores the non-zero elements.But if I use >> sparse(zeros(100)), no element should be stored for this case in the memory after the whole operation is completed as there is no non-zero element in zeros(100) (In this case, the number of elements matlab has to handle is quite reasonable, so there is no question of running out of memory!). The memory consumption for this operation (>> sparse(zeros(100))) should have been negligible as matlab practically does not store anything at the end. But, I noticed that matlab eats up quite a bit of memory for this. I would like to know the reason. Thank you again.

Ken Atwell 2014-4-20

See Geoff's comments below. While even an all-zeros sparse matrix will occupy some memory for book-keeping, it should be minimal in the grand scheme of things.

请先登录，再进行评论。

Answer 2

Geoff Hayes 2014-4-19

3
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/126295-memory-usage-in-sparse-matrix#answer_133853

在 MATLAB Online 中打开

Hi tanmoy,

The documentation for this function states something like the following (when you run help sparse from the command line):

sparse Create sparse matrix.
S = sparse(X) converts a sparse or full matrix to sparse form by squeezing
out any zero elements.

So you're right - the result of sparse on this matrix should be some kind of empty matrix since all elements are zero and so have been squeezed out. But before sparse is even called, zeros(80000) is invoked which will try to create a 80000x80000 matrix of zeros. So this is probably where you are getting stuck (and I noticed the same behaviour on my computer with your example). There is nothing wrong with the call to sparse , it is just that it follows the attempted creation of a huge matrix that chews up a lot of memory.

Hope that this helps!

Geoff

3 个评论
显示 1更早的评论隐藏 1更早的评论

tanmoy 2014-4-19

Thank you for clarifying my queries.The answer is really illustrative. However, I have one more confusion regarding this.

As I understand, a sparse() matrix in matlab only stores the non-zero elements.But if I use >> sparse(zeros(100)), no element should be stored for this case in the memory after the whole operation is completed as there is no non-zero element in zeros(100) (In this case, the number of elements matlab has to handle is quite reasonable, so there is no question of running out of memory!). The memory consumption for this operation (>> sparse(zeros(100))) should have been negligible as matlab practically does not store anything at the end. But, I noticed that matlab eats up quite a bit of memory for this. I would like to know the reason. Thank you again.