Program Optimisation: Out of Memory

Question

John 2011-7-29

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/12655-program-optimisation-out-of-memory

I have a program that is on the limit of using all the available memory. I have done all I can to increase the memory limit but because the student edition is 32-bit the limit is 2Gb (correct me if I am wrong please). Below is >>memory when started up.

Maximum possible array:            2046 MB (2.146e+009 bytes) *
Memory available for all arrays:   3275 MB (3.434e+009 bytes) **
Memory used by MATLAB:              474 MB (4.974e+008 bytes)
Physical Memory (RAM):             3882 MB (4.071e+009 bytes)
*  Limited by contiguous virtual address space available.
** Limited by virtual address space available.

Therefore I need to optimise my code to work within these small memory limits. The error comes from here:

??? Error using ==> times
Out of memory. Type HELP MEMORY for your options.
Error in ==> couple at 58
        k = idx1.*100e6 + idx2.*10e6 + idx3.*10 +
        idx4.*10e6 + idx5.*100e6;

The size of idx is (2.6e5, 200). Is there any way to optimise the code to allow a larger matrix to be computed. The matrix has been generated from ODE45, however, reducing the accuracy even further is undesirable.

Thank you for your assistance.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Oleg Komarov 2011-8-3

I usually have 1.1 GB of maximum possible array at startup (2011a Win32 Vista) and a dataset of ~450 MB.

I had to split the dataset into a structure and saved it with -struct and -v6 option.

Then I wrote a function that loops a handle through each portion of the struct.

It takes a little to understand how to operate on parts rather than on the entire dataset, but if you think it may be helpful I will share my solution.

Also, you can save the dataset in binary format and fread it with the approariate offsets (alternatively to save with -struct option).

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

John 2011-8-27

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/12655-program-optimisation-out-of-memory#answer_19864

在 MATLAB Online 中打开

I solved the problem by making my ODE settings to record fewer time intervals:

tspan = [0 a];
tspan = 0:0.01:a;

It still has the same accuracy but holds fewer numbers.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Titus Edelhofer 2011-7-29

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/12655-program-optimisation-out-of-memory#answer_17284

Hi,

hmm, somehow this looks strange: the size of idx is (2.6e5, 1)? That means, that one of your idx vectors needs about 2MByte of memory. Doesn't sound too much to me ...???

Titus

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

John 2011-7-29

Sorry, each idx is (2.6e5, 200).

请先登录，再进行评论。

Answer 3

Bjorn Gustavsson 2011-7-29

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/12655-program-optimisation-out-of-memory#answer_17286

Maybe you can cut up your problem and work with subsets of idx in sequence. This might not be "ideal" but if nothing else works it might get the job done...

5 个评论
显示 3更早的评论隐藏 3更早的评论

Bjorn Gustavsson 2011-7-30

OK, if you can't use this subset approach in either dimension, then it looks dire. Maybe you can use memmapfile to store all your big matrices. It might be a workaround - but I haven't used it so I have no grip on if it would work or if there is other pitfalls.

Walter Roberson 2011-7-30

memmapfile requires virtual memory equivalent to the size of the data stored.

请先登录，再进行评论。

Answer 4

Walter Roberson 2011-7-29

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/12655-program-optimisation-out-of-memory#answer_17336

The limit for the student edition is the same as the limit for the professional edition.

2 个评论
显示无隐藏无

John 2011-7-30

From what I read on the MathWorks website only 32-bit editions have the 2Gb limit and 64-bit editions have a much higher limit. Can you confirm that I miss-read this?

Walter Roberson 2011-7-30

The 32 bit versions (student _or_ professional) have a limit of 32 bits of address space, which is 4 Gb total including all memory required for system interfaces.

The 64 bit versions (student _or_ professional) in theory could have a limit of 64 bits of address space, but in practice current processors use no more than 48 bits of address space. I do not know how MATLAB divides up that virtual address space. I _think_ I saw a hint that _one_ of the MATLAB library calls was effectively restricted to 40 bits worth of addressing, but I could easily be wrong about that.

Anyhow, student version and professional version are exactly the same memory limits and have been since the first version that was actually called "Student Version". The original two Springer-Verlog books that included a student matlab *did* have lower memory limits, but it was called "Student Edition". (And yes, I do continue to get "Edition" vs "Version" confused myself.)

请先登录，再进行评论。

Answer 5

Walter Roberson 2011-7-30

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/12655-program-optimisation-out-of-memory#answer_17382

在 MATLAB Online 中打开

In-place operations might help:

k = idx1.*100e6 + idx2.*10e6 + idx3.*10 +
      idx4.*10e6 + idx5.*100e6;

could become

k = idx1;
k = k + idx5;
k = k .* 10;
k = k + idx2;
k = k + idx4;
k = k .* 10e5;
k = k + idx3;
k = k .* 10;

This would not require any intermediate vectors, just the output vector that is necessary anyhow. In theory any relatively recent MATLAB should be able to handle

A = A + B

by overwriting A as the computation goes along.

2 个评论
显示无隐藏无

John 2011-7-30

Thanks for your post. I now get the error on the first line:

??? Error using ==> times

Out of memory. Type HELP MEMORY for your options.

k = idx1.*100e6;

Walter Roberson 2011-7-30

I didn't start with k = idx1.*100e6 -- I started with k = idx1 .

None the less, if your idx1 is so big that another array the same size cannot be created, then there is really not much you can do about the situation.

You could do hacks such as saving the variables to individual .mat files, and then using system() to run a matlab routine that loaded just enough at a time to take one more step, clearing the variable as soon as it was done with, and then save()'ing the result to a .mat file. What this would get you is that in the second matlab session, the _only_ thing in your workspace would be MATLAB itself, the standard I/O libraries, and the variables you were working with: the effect would be to gain virtual address space.

If you are willing to do some mex'ing, you could probably create a pipe() over to a MATLAB session to do the calculation for you. Note that pipe() might need to be coded differently between Windows and the unix-based systems; in _theory_ from Windows XP SP2 and later the code could be compatible, but I have seen too many hints that the POSIX module built in to Windows doesn't really deliver POSIX.

请先登录，再进行评论。

Answer 6

Jan 2011-7-30

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/12655-program-optimisation-out-of-memory#answer_17399

在 MATLAB Online 中打开

If the values of idx and k match into a UINT32, use this type to reduce the memory consumption to 50%:

idx1 = uint32(...);
idx2 = uint32(...); ... etc
k = idx1.* uint32(100e6) + idx2.* uint32(10e6) + idx3.*10 + tc...

This together with Walter's inplace method can reduce the needed memory efficiently.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

John 2011-8-3

Thank you for your reply. I tried it and it ran much faster. However, how do I multiply the k vector with another vector which is not uint32? For example:

ans = diff.* k;

k is a vector (n,1) and diff is a vector (n,1) and I wish to multiply each element only e.g. k(n-1,1)*diff*(n,1), k(n,1)*diff(n,1) ...etc.

请先登录，再进行评论。