large sparse matrices fail with accumarray

1 次查看（过去 30 天）
Balint Takacs
Balint Takacs 2012-3-20
Large sparse matrices are easily created by 'sparse':
a = sparse(1e6, 1e6);
is working fine. However,
b = accumarray([1 1], 1, [2^31 1], @sum, [], true);
fails with "maximum variable size allowed by the program is exceeded." Matrix sizes with less than 2^31 elements are going through.
Why is this stupid limitation and how to circumvent it? I need accumarray to work on matrices of that size. The solution must be fast.
Using R2007a 32-bit Linux.
Cheers,
Balint

请先登录，再回答此问题。

回答（2 个）

the cyclist
the cyclist 2012-3-20
I believe this particular memory issue is related to the addressing of elements of the array. This limitation is mentioned (although not discussed in detail) in Section 1.5 of this guide to memory use: http://www.mathworks.com/support/tech-notes/1100/1107.html. There is also discussion in this document: http://www.mathworks.com/support/tech-notes/1100/1106.html
  3 个评论
显示 1更早的评论
the cyclist
the cyclist 2012-3-20
Are you able to upgrade to a more recent version? This statement from the documentation:
"Note that in MATLAB 7.4 (R2007a) and later you can create arrays with more than 231-1 elements but this was not officially supported until MATLAB 7.5."
suggests to me that your version was a bit transitional as they implemented the new functionality. For example, in R2012a I am able to execute the statement
b = accumarray([1 1], 1, [2^47 1], @sum, [], true);
without error.
Balint Takacs
Balint Takacs 2012-3-21
Thank you. Unfortunately I have no option to upgrade.
I guess the problem is related to accumarray creating some temporary, which is limited by this bound you are referring to.
I have solved my problem by slightly changing the data structures, and using sparse(i,j,val,m,n) instead accumarray.

请先登录，再进行评论。

Richard Crozier
Richard Crozier 2012-5-21
I'm not sure how your two statements here are equivalent?
a = sparse(1e6, 1e6);
creates a sparse matrix of all zeros of dimensions (1e6, 1e6), making only 1e12 elements, and nothing needs to be allocated since there are no non-zero values
b = accumarray([1 1], 1, [2^31 1], @sum, [], true);
Creates a sparse vector of size (2^31, 1) with a single nonzero value allocated at (1,1). The equivalent call to this is surely:
>> a = sparse(1, 1, 1, 2^31, 1)
which fails with:
??? Error using ==> sparse Sparse matrix sizes must be non-negative integers less than MAXSIZE as defined by COMPUTER. Use HELP COMPUTER for more details.
Which is probably the same error in another guise.
EDIT:
in fact even
a = sparse(2^31, 1)
fails with this error.
  2 个评论
Balint Takacs
Balint Takacs 2012-5-21
Both of your lines execute without error on my MATLAB, while the accumarray one does not.
Richard Crozier
Richard Crozier 2012-6-14
You're right by the way, it does indeed have this stupid limitation, I've now come across it elsewhere. Not sure why it doesn't happen in the exact case above on my machine though.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Sparse Matrices 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by