Dynamic allocation of mxArray structure

3 次查看（过去 30 天）
Fedor Iskhakov
Fedor Iskhakov 2017-6-22
评论： Fedor Iskhakov 2017-6-23
Hi everyone,
I have a mex function in C that internally generates an unpredictably large volume of data to be output back to Matlab. I would like the return to be a Struct Matrix of the 1 by N, where N is the unknown large number. The memory should be allocated by chunks, so that in case of insufficient memory the calculation can be stopped and the return that does fit into memory handled safely back to Matlab.
It seems that the combination of mxSetN() and mxRealloc() is a proper combination, but I can't find sufficient info on how to use mxRealloc to allocate space for additional elements of structural matrix.
Here is a minimal code example of what I'm trying to do
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
mwSize i;
mxArray *mxp;
nlhs[0]=mxCreateStructMatrix(1,1,0,NULL); //structure with one element and no fields for now
while (continue calculation)
{
//one more round of calculations that creates a lot of output
//stored in structure g with double* field1,..,K
mxAddField(nlhs[0],"field1");
mxp=mxCreateDoubleMatrix(appropriate,size,mxREAL);
if (mxp==NULL) {mexErrMsgTxt("Failed to allocate memory!")}
mxSetField(nlhs[0],i,"field1",mxp);
...
mxAddField(nlhs[0],"fieldK");
mxp=mxCreateDoubleMatrix(appropriate,size,mxREAL);
if (mxp==NULL) {mexErrMsgTxt("Failed to allocate memory!")}
mxSetField(nlhs[0],i,"fieldK",mxp);
i++;
mxSetM(nlhs[0],i)
//What then?
}
}

请先登录，再回答此问题。

采纳的回答

Jan
Jan 2017-6-22
Letting an array grow iteratively consumes a lot of resources. Therefore it is a good idea to create the new data in chunks and reply them separately.
As James said already, mxCreateDoubleMatrix stops the Mex function immediately if it is not successful. Then all formerly creatzed data are lost. A workaround would be to call the Mex function for each junk in a loop and collect the data in Matlab:
allChunk = cell(1, 1000);
iChunk = 0;
while ~ready
try
newChunk = myMexfunction();
iChunk = iChunk + 1;
allChunk{iChunk} = newChunk;
if iChunk == numel(allChunk) % Container exhausted
allChunk{iChunk + 1000} = []; % Expand
end
catch
disp('Memory exhausted');
ready = true;
end
end
But if te MEX function could not allocate any memory anymore, Matlab is in danger to crash also. You cannot do a lot of computations anymore, most of all you cannot combine the chunks to a large array.
Perhaps it is better to write the junks to the disk (SSD preferrably), such that an exhaustion is less likely and the memory is not filled at all. This would be much better than a strategy which is expected to cause an out-of-memory crash.
  1 个评论
Fedor Iskhakov
Fedor Iskhakov 2017-6-23
Thanks, saving to disk does sound like most reasonable thing I can do. Running the loop in Matlab would also be more problematic in my application..

请先登录，再进行评论。

更多回答（2 个）

Philip Borghesani
Philip Borghesani 2017-6-22
编辑：Philip Borghesani 2017-6-22
Although the code example you linked to will do the resizing operation on a structure and in general I like it better then the code you posted. I suggest not ever resizing the main structure but allocating it to some reasonable max size from the beginning. Resizing and reallocating operations are slow and tend to cause memory fragmentation issues. In addition structure arrays have quite a bit of memory overhead proportional to the dimensions of the structure.
Hopefully (and if not try to fix it) the amount of memory used by field1 is relatively large compared with the size of your overall structure. If size is a few thousand or larger then the memory used to store the basic structure is insignificant and it can be allocated to it maximum size initially. Your function can return the max "i" reached as a second output or the calling code can just check for empty fields when processing the data and stop.
  1 个评论
Fedor Iskhakov
Fedor Iskhakov 2017-6-23
I take your point, thanks, I'm aware that I'm looking for a "second best" here. The output format can not be changed in the short term, but in the longer term we should reconsider this whole setting.

请先登录，再进行评论。

James Tursa
James Tursa 2017-6-22
编辑：James Tursa 2017-6-22
"... The memory should be allocated by chunks, so that in case of insufficient memory the calculation can be stopped and the return that does fit into memory handled safely back to Matlab. ..."
You can't do this in a mex routine.
In these lines (and other similar lines in your code):
mxp=mxCreateDoubleMatrix(appropriate,size,mxREAL);
if (mxp==NULL) {mexErrMsgTxt("Failed to allocate memory!")}
mxp will never be NULL and you will never get inside the if-block. Inside a mex routine, if there is an API call that results in an out of memory condition, the mex routine will immediately return control to the caller with an error. In that situation it will never execute the next line in your code (i.e., you will never return from the mxCreateDoubleMatrix function call). The only way you actually get a NULL value from API function calls in your code that allocate memory is if they are used in an Engine application ... it never will happen in a mex routine.
  1 个评论
Fedor Iskhakov
Fedor Iskhakov 2017-6-23
Thanks for pointing this out. I did notice that mxRealloc() behaves exactly like this, but I was sure the check for NaNs returned from mxCreate** routines worked before to detect memory shortage. What a bummer!

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Programming 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by