Peformance issue while fetching data from C Mex function.

Question

Manju gurik 2022-6-8

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1736070-peformance-issue-while-fetching-data-from-c-mex-function

评论： Manju gurik 2022-6-8

Hi,

We have C mex funciton that is implemented to fetch data from external source over network and fill this data in C mex function(using mxCreateDoubleMatrix and copy this data into plhs array), when this data is huge(around 22 MB), accessing this data on matlab is very slow takes while(20 seconds), if we chunk this data(in 100Kbs), data is copied and accessed on matlab variable faster. I wanted to know, is there memory limit while allocating and copying data and access into matlab variable. Appreciate your suggestions.

Thank you'

2 个评论
显示无隐藏无

Jan 2022-6-8

Without seeing the code it is impossible to guess, if it contains an avoidable bottleneck. 22MB is not considered as "huge" usually. "100Kbs" is a strange unit and it is not clear, how this could define a chunk.

The memory limit is defined in Matlab's preferences => WorkSpace => Array size limit

Manju gurik 2022-6-8

Thank you for the reply. here is the code.

Mex Function: ReadRows.c

void mexFunction( int nlhs, mxArray *plhs[],

int nrhs, const mxArray *prhs[] )

{

size_t cnt, num;

int OutputIndex = 0;

double ***pFields, **pTmp;

double ConnNumber, ReadIndex, NumFields, NumtoRead, *InputFieldSize, MaxtoRead;

double *ReadStatus, *NumRead, **OutputFieldSize, **BitFieldStatus;

size_t iFields, iRead;

ConnNumber = mxGetScalar(prhs[0]);

ReadIndex = mxGetScalar(prhs[1]);

NumFields = mxGetScalar(prhs[2]);

InputFieldSize = mxGetPr(prhs[3]);

NumtoRead = (int)mxGetScalar(prhs[4]);

plhs[OutputIndex] = mxCreateDoubleMatrix(1, 1, mxREAL);

ReadStatus = mxGetPr(plhs[OutputIndex++]); /* Read status */

*ReadStatus = 1; // ADI_STAT_FAILURE

plhs[OutputIndex] = mxCreateDoubleMatrix(1, 1, mxREAL);

NumRead = mxGetPr(plhs[OutputIndex++]); /* Number Read */

*NumRead = 0;

iFields = (size_t)NumFields;

NumFields = iFields;

// limit to left hand side inputs

MaxtoRead = (size_t)( ((double)nlhs-2.0) / (NumFields+2.0) );

if ( NumtoRead > MaxtoRead )

NumtoRead = MaxtoRead;

QueryNumberOfRows( ConnNumber, &MaxtoRead );

if ( NumtoRead > MaxtoRead )

NumtoRead = MaxtoRead;

iRead = (size_t)NumtoRead;

NumtoRead = iRead;

if ( nlhs < 2+NumtoRead*(NumFields+2) )

mexErrMsgTxt("Need outputs: Status, NumRecsRead, [BitStat, OutputSize, NumFieldsOfData...]*NumtoRead");

if ( NumtoRead > 0 )

{

BitFieldStatus = (double**)mxMalloc( (mwSize)2*iRead*sizeof(double*) );

OutputFieldSize = BitFieldStatus + iRead;

pFields = (double***)mxMalloc((mwSize)iRead*sizeof(double**) );

// since iFields usually small compared to iRead, allocate block

pTmp = (double**)mxMalloc((mwSize)iRead*iFields*sizeof(double*) );

}

for(num = 0; num < (size_t)NumtoRead; num++)

{

/* Output arguments */

plhs[OutputIndex] = mxCreateDoubleMatrix((mwSize)iFields, 1, mxREAL);

BitFieldStatus[num] = mxGetPr(plhs[OutputIndex++]); /* Read bit field status */

plhs[OutputIndex] = mxCreateDoubleMatrix((mwSize)iFields, 1, mxREAL);

OutputFieldSize[num] = mxGetPr(plhs[OutputIndex++]); /* Field size read */

// pFields[num] = (double**)mxMalloc( iFields*sizeof(double*) );

pFields[num] = pTmp + num*iFields;

/* Output Field arguments */

for(cnt = 0; cnt < NumFields; cnt++)

{

plhs[OutputIndex] = mxCreateDoubleMatrix((mwSize)InputFieldSize[cnt], 1, mxREAL);

pFields[num][cnt] = mxGetPr(plhs[OutputIndex++]);

}

while( OutputIndex < nlhs )

{

plhs[OutputIndex++] = mxCreateDoubleMatrix(0, 0, mxREAL);

}

FetchData(ConnNumber, ReadIndex, NumFields, NumtoRead, InputFieldSize, ReadStatus, NumRead, BitFieldStatus, OutputFieldSize, pFields);

}

Matlab Calling:

chunkSize = 50000;

num_cols = NumFields + 2; %(3 data fields + 2(bitfieldstatus and outputsize)

for k = 0:chunkSize:TotalNumRows

startIndex = k;

dataStartIndex = startIndex + 1;

dataEndIndex = startIndex + chunkSize;

if (dataEndIndex > TotalNumRecords)

dataEndIndex = TotalNumRecords;

chunkSize = TotalNumRecords - startIndex; % Last chunk remaining records that are less than chunk size(2048)

clear varargout;

end

[varargout{1:(NumFields + 2)*chunkSize+2}] = ...

ReadRows(conNum, startIndex, NumFields, InputFieldSize, ...

chunkSize);

data_A(dataStartIndex:dataEndIndex) = [varargout{1, 5:num_cols:end}]';

data_B(dataStartIndex:dataEndIndex) = [varargout{1, 6:num_cols:end}]';

data_C(dataStartIndex:dataEndIndex) = [varargout{1, 7:num_cols:end}]';

end

In the above code, when we use chunkSize = 2048, data assignments to data_A, data_B, data_C is immediate, if we use chunkSize = 50000, it will take a while(30 seconds) to reach to this line

data_A(dataStartIndex:dataEndIndex) = [varargout{1, 5:num_cols:end}]';

even though mex function returns immediatly afte fetching data from FetchData() which takes less than a second.

My question, why its taking time to copy/transfer the data after mex function ReadRows call when we have bigger chunks. Is there any way to improve this.

Here all the data fields are double values(vector).

Thank you.

请先登录，再进行评论。

请先登录，再回答此问题。