How to quote variables whose names are in order in a for loop?

Question

Jason Yang 2021-3-29

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/786471-how-to-quote-variables-whose-names-are-in-order-in-a-for-loop

评论： Steven Lord 2021-4-27

Hi! I want to store a 6D tensor A which is so large that I meet the 'out of memory' error. So I want to store it by parts. i.e. store A by sub-tensors A1,A2,...An. I know I can create variables whose name are in order by eval(). For example, I can create x1,...,x10 by:

VarNum = 10; 
vars = []; 
for i = 1:VarNum
    eval([['x',num2str(i-1)],'=','i',';']); 
end

But I don't know how to quote x1,...x10 in a for-loop when I don't store x1,...x10 into an array or cell which contains too large space and I will encounter 'out of memory' error. Can anyone help me? Thank you!

7 个评论
显示 5更早的评论隐藏 5更早的评论

Walter Roberson 2021-4-7

When MATLAB asks the operating system for memory, the operating system looks around for unused physical "pages". In theory there are a few different possible physical page sizes, but in practice for Windows it will almost always be 4 kB. That is, the physical memory is broken up into chunks of 4096 bytes and those are allocated as chunks instead of trying to keep track of every byte individually.

So, the operating system will figure out how many 4 kilobyte chunks of memory are needed, and will find them somewhere (or make them available.) Then it will put the addresses of all of those physical pages together into a translation table that matches between consecutive logical addresses and corresponding physical addresses. The program (MATLAB) uses consecutive numeric addresses for the chunk of memory, and the hardware works to convert the numeric addresses into physical addresses.

As far as the program is concerned, it just sees all the memory in one array as if it were adjacent physical chips. But for a different array, it might have been told a quite different numeric base address. Which might be a kind of fiction too -- the second array might perhaps have ended up in physical memory in chips immediately beside the ones for the first array. Or not. MATLAB asked for memory, the operating system says "Here is a base address, use this base address plus a linear offset and the hardware will take care of accessing the right scattered physical memory pages". MATLAB asked for more memory, the base address might be quite "far" from the first array, no matter where the physical memory used is.

It is meaningful to talk about chunks of contiguous physical memory -- physical memory pages whose chips are right beside each other. For most purposes, the operating system does not need contiguous physical memory... but if it can happen to get it, then that can reduce overhead a little. (Contiguous physical memory is sometimes required for talking to peripherals.) In some cases of a system being fairly loaded with programs and chunks of memory, that reduction of overhead can turn out to be important; for example the hardware might only be able to track 65536 different chunks of memory at the same time, so it might become important to use consecutive physical blocks so that the hardware can say "this page and the next three" instead of listing the pages individually. There is some use to it... but until the system gets overloaded in that way, the system doesn't care how scattered the physical memory pages are for a program.

For the purposes of user-level programs, they only care that they are given a base address and can access any location in the chunk by using the basic address and a linear offset -- if byte offset 4097 is associated with a physical memory page that is not the "next" physical memory page after byte offset 4095, then that is taken care of in hardware. Any one array of the low-level data types (uint8, uint16, uint32, uint64, int8, int16, int32, int64, single, double, logical, char) is taken care of at the MATLAB level in terms of start address and offset, with the hardware taking care of the physical details. The addresses for any one numeric array are all contiguous at the "virtual" memory level.

When you use cell arrays or structs or any other composite data type, then the different entries are not specifically allocated as being "beside" or even "close to" the other entries at the composite level. The structure entry abc(1).def might be allocated some distance away from abc(2).def or abc(1).ghi -- but all of the numeric array abc(1).def is contiguous virtual memory within itself.

Why does this all matter in the situation?

Well, when you have limited amounts of available memory, then depending on what has happened before in the program, you might encounter a situation where there is not enough available consecutive virtual memory to hold (say) a 6 gigabyte block, but there might be enough consecutive virtual memory to hold six 1 gigabyte blocks. The total amount of available virtual memory can, in some cases, be less important than how fragmented it is. There are definitely situations where if you had allocated that 6 gigabyte block "first thing" in a program that it would have given it to you and then arranged everything else to fit, but if you ask for it later in the program it might tell you it cannot do it because there just isn't enough in one piece at the moment.

Thus, Stephen was suggesting that as long as you have enough available memory to hold the total amount of data, that using a cell array might be a way to split the requirements into smaller pieces that are more likely to individually fit where one big chunk did not fit.

Using a cell array will not help in the situation where the total amount of data to be loaded is larger than can fit in your system, In those kinds of situations, using tall() arrays or using datastore object can sometimes help.

Jason Yang 2021-4-27

@Walter Roberson That helps a lot! My question is solved. Thank you very much!

Steven Lord 2021-4-27

An analogy that might make this clear is that of a hardcover version of the combined Lord of the Rings series of books. That volume requires a certain width of shelf space to store, and all that space has to be on the same shelf.

If you were to split the series into three volumes they would take up (basically) the same total width, but you could put the first volume on one shelf and the other two on the next shelf down. Though if you misplace volume 2 it could take you a while to find it when you want to read from it.

The combined hardcover is a double array, the three volumes are a cell array each cell of which contains part of the larger double array.

请先登录，再进行评论。

请先登录，再回答此问题。