speed up computation of huge matrix?

Question

Leon 2011-10-14

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/18287-speed-up-computation-of-huge-matrix

Hi,

I need to compute the mean of a huge matrix (20000*20000*time) at several times, Is there any way to speed up the computation when calculating this?

Cheers

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Jan 2011-10-14

2
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/18287-speed-up-computation-of-huge-matrix#answer_24569

Yes, or no. It depends.

An arry of the size [20000 x 20000 x time] occupies 3.2 GB, if time is 1 and for type DOUBLE. If time is larger, more memory is needed - in contiguous free blocks.

Calculating the MEAN over the 1st dimension is much faster than over the 2nd one, because the memory is processed in contiguos blocks.

Using INT16 values would reduce the memory footprint, but the values must be integer then.

Without any more detailed description, a more specific answer is impossible.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Edric Ellis 2011-10-14

2
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/18287-speed-up-computation-of-huge-matrix#answer_24580

As Jan points out, this is very large data. One option is to use distributed arrays with Parallel Computing Toolbox (and MATLAB Distributed Computing Server to spread the computation over the memory of multiple machines). But it may be better yet not to have all of your data in memory at one time.

If you have a capable GPU, you can use gpuArrays. On my NVIDIA C1060, I can calculate the mean of 20000x20000 doubles in 0.04 seconds vs. 0.2 seconds on my CPU (R2011b). A more recent Tesla card such as the C20xx family would probably out-perform this.

5 个评论
显示 3更早的评论隐藏 3更早的评论

Edric Ellis 2011-10-17

Building a 3.2GB gpuArray from a CPU array takes about 2.0 seconds on my system. This is why it's important to keep data on the GPU as long as possible, and preferably build it there (there are zeros/ones etc. functions which can build a 3.2GB gpuArray in 0.025 seconds; rand on the GPU is somewhat slower, but still quicker than building on the host and transferring).

Jan 2011-10-17

@Edric: Thanks for this summary. I've upgraded from a Pentium-M to a dual-core some month ago. Now my experiments with mutli-threading are less abstract. CUDA is the next step.

请先登录，再进行评论。