preallocate array without initializing

23 次查看（过去 30 天）
Ian
Ian 2018-11-30
评论： Walter Roberson 2018-12-2
Is there any way to preallocate a matrix without initializing it to either NaN's, zeros or ones?
I'm working with large image data, typically 1-4 billion pixels, and need to preallocate the array so I can read the data in in chunks. I don't need the array initialized because I will either fill the entire array if reading the data is successful, or throw an exception if it fails.
Preallocating the array with either zeros or NaN's takes matlab several seconds to initialize the array.
Allocating a large array like this in C++ returns immediately because it neither new(...) or malloc(...) bother to initialize the memory.
Is it possible to allocate the array either in a C++ mex file or using coder.ceval(...) to avoid the initialization time?

请先登录，再回答此问题。

采纳的回答

Steven Lord
Steven Lord 2018-11-30
If you want a matrix of class int16 filled with 0 elements, don't wrap a call to zeros in a call to int16. Instead use the typename input listed on the documentation page for the zeros function.
z = zeros(1e4, 1e5, 'int16');
Are you trying to read in a sequence of images that may take up more memory than you have avaialble (or a significant fraction of your available memory?) If so consider creating a tall array from an ImageDatastore and performing your analysis on the tall array. See this documentation page for more information on tall and datastore arrays.
  8 个评论
显示 6更早的评论
Ian
Ian 2018-12-2
编辑：Ian 2018-12-2
Which only shows how gray my beard is! Thanks for your comment. This was not a hardware feature of either the memory or the processors in the Z-80 and intel 8080 based systems I cut my teeth on.
For anyone like me both clueless about and interested in what "Demand-Zero-Paging" is and why it is important, here's a good brief description, in answer to a question about the concept:
http://linux.derkeiler.com/Newsgroups/comp.os.linux.development.system/2006-11/msg00176.html
If I understand it correctly, a request for a large block of zero'd memory can be memory-mapped to a single small segment of physical memory, and only redirected to a larger physical address space when actually written to.
That would explain why a call to a large quantity of zeros(...) is much faster than a call for nan(...) or ones(...).
Walter Roberson
Walter Roberson 2018-12-2
Some history . I have not found anything definitive as to when Demand Zero was introduced . Demand paging dates to 1961.
Demand Zero requires MMU which were not present in the intel 80* series until the 80286
https://sigops.org/sosp/sosp15/history/04-satya-slides.pdf
https://homes.cs.washington.edu/~arvind/cs422/lectureNotes/l16-2.pdf

请先登录，再进行评论。

更多回答（4 个）

Bruno Luong
Bruno Luong 2018-11-30
编辑：Bruno Luong 2018-11-30
MEX can do that, use for example this utility
per isakson
per isakson 2018-11-30
编辑：per isakson 2018-11-30
Try this
>> A(100,100) = 0;
>> whos A
Name Size Bytes Class Attributes
A 100x100 80000 double
I believe it is described by UndocumentedMatlab, but don't find it now. However, see the two first comments to the blog piece, which I linked to.
Maybe, more relevant
>> B(100,100)=uint8(0);
>> whos B
Name Size Bytes Class Attributes
B 100x100 10000 uint8
Ian
Ian 2018-11-30
A little experimenting has determined the following:
z = zeros(m,n);
is apparently quite fast, but only helpful if one needs a matrix of doubles.
clear z; tic(); z=zeros(10000,100000); toc()
Elapsed time is 0.000367 seconds.
>> whos z
Name Size Bytes Class Attributes
z 10000x100000 8000000000 double
>> clear z; tic(); z=int16(zeros(10000,100000)); toc()
Elapsed time is 5.164922 seconds.
>> whos z
Name Size Bytes Class Attributes
z 10000x100000 2000000000 int16
Unfortunately, converting the result to int16 is very slow and presumably uses about 5x final memory requirements in the process, which if the array is large, will cause problems.
PI's approach probably avoids creating the matrix first as doubles, but is also slow (but faster than the above):
>> clear z; tic(); z(10000,100000) = int16(0); toc()
Elapsed time is 1.023650 seconds.
>>
>> clear z; tic(); z(10000,100000) = 0; toc()
Elapsed time is 5.692720 seconds.
Ian
Ian 2018-11-30
编辑：Ian 2018-11-30
BL is apparently correct -- it can be done in a MEX file -- and I have located a functional solution on Matlab File Eschange: 31362-uninit-create-an-uninitialized-variable-like-zeros-but-faster
It is a self-compiling MEX file which allows creation of matrices of any data type without initializing them.
clear z; tic(); tic(); z=uninit(23040,46080,'int16'); toc()
Elapsed time is 0.000231 seconds.
This solution is old (last updated 2011), but works in R2018a on MacOS and on Linux under R2017b.
It does leave the resulting matrix uninitialized.

类别

在 Help Center 和 File Exchange 中查找有关 Performance and Memory 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by