Working with a very large matrix

7 次查看（过去 30 天）
sh as
sh as 2016-4-4
评论： John D'Errico 2016-4-8
I would like to make a large matrix of size ~20000000 * 20000 and working with it. But it is not allowed because of memory shortage. as most of my elements at first, are zero, I could make the matrix with sparse function, but then my matrix will be filled with all elements of nonzero. the value of elements at first are 0-9 natural numbers and then real number (with 2 digit precision is fine).
Does anyone have any idea how to deal with such a large matrix? or any alternative way (data structure) to work with?
p.s: somebody suggested "parallel computing server", but I don't have it on my Matlab.
Thank you very much.

请先登录，再回答此问题。

回答（1 个）

John D'Errico
John D'Errico 2016-4-4
You cannot specify that a given number will only have 2 decimal places stored. If floating point, it must be either a single or a double.
Your array has 2e7*2e5=4e12 elements in it. Stored as single precision numbers, that is 4 bytes per element. So a total of roughly 1.6e13 bytes of RAM. Roughly 16 terabytes. Do you have that much memory?
If there are only a TINY percentage of non-zeros, then yes, you CAN use sparse storage, as long as you create the array using the sparse function, without creating a full array first.
If the array will undergo fill-in with further manipulations, then sparse storage is not acceptable.
  2 个评论
sh as
sh as 2016-4-8
编辑：sh as 2016-4-8
Yes, and I am looking for a solution. Do you think if I can make the matrix and the process on it with C/C++ code? (I mean using mex file). or it does change anything?
Thanks,
John D'Errico
John D'Errico 2016-4-8
Of course, IF you can create the matrix (in sparse form) and pass it along to use it in C/C++, then you can then do any processing on it that you wish, as long as you do not exceed your memory limits. Since I have no idea what processing you plan on doing, I cannot say that you can or cannot succeed. All of that depends on the degree of fill-in you have due to your processing.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Logical 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by