Data access in sparse matrix

Question

Paul Baxter 2012-10-10

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/50403-data-access-in-sparse-matrix

I am trying to speed up a section of code which uses a very large (300,000 x 8 x 8) matrix whose elements are derived from other variables via a non-trivial function. Currently I have pre-calculated all the terms and saved the matrix, but even loading this matrix into memory takes a long time.

In any individual run of my code it's likely that I will only need to access a small subset of the values (probably 5000 of the 300,000 matrices?), but those terms will need to me accessed many times (1000's), so I was considering changing to using a sparse matrix and calculating the terms as required, if not previously calculated.

To decide if this is a worthwhile change, can someone tell me how the sparse memory access works in MATLAB. Specifically, if I already have 4999 of the matrix slices calculated, will the following be very slow operations: a) search to see if a given matrix has already been computed and placed in the sparse matrix b) place a new matrix in the sparse matrix.

b) is the one I don't really understand - I suspect I can solve any a) problems with a large boolean vector or similar.

If it makes a difference this code has to run on R2010b.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

James Tursa 2012-10-10

编辑：James Tursa 2012-10-10

FYI, MATLAB does not directly support 3D sparse arrays. To get that behavior you have to use something like Matt J's ndSparse class:

http://www.mathworks.com/matlabcentral/fileexchange/29832-n-dimensional-sparse-arrays

Also, it is typically more efficient to store the 2D slices in the first two indexes rather that the last two. E.g., have the array 8 x 8 x 300,000.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Doug Hull 2012-10-10

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/50403-data-access-in-sparse-matrix#answer_61523

The truth is, a test is worth a thousand expert opinions.

Have you run this through the profiler to confirm that it is the matrix access that is killing your speed? Break down any slow lines of code into smaller pieces so you can see which operation in a line of code is hurting you as profiler works on a line by line basis.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Paul Baxter 2012-10-11

I had two code versions - the first did all the calculations of the numbers required on the fly, runtime ~1000 seconds 80% on the calculations. The second loads in the pre-calculated array (a full matrix) taking about 50 seconds to do and then runs much faster (220 seconds). The problem with this version is that occasionally I have to load in a different set, and the code has to run smoothly in real-time (so no 30 second hang-ups)

I'll code up the sparse approach and profile that as well.

请先登录，再进行评论。

Answer 2

Matt J 2012-10-10

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/50403-data-access-in-sparse-matrix#answer_61541

编辑：Matt J 2012-10-10

I can't imagine that accessing the matrix is slowing you down appreciably on any decent machine. It might help a bit if you reorganize your array to be 8x8x300000, because the first dimensions of an array is always the slowest to access. I'm also surprised that loading it takes a long time, though I'm not sure what "long" is for you.

As for sparse matrices, updating them on the fly is always a lot slower than for full matrices, but it doesn't sound like it matters. Only a small percentage of your operations sound like they're actually devoted to generating and filling the 'slices'. Most of your computation involves pulling them out again, which should be very fast no matter what you use.