Matlab Engine Array Transfer Size Limitation

Question

0 个投票

I am using engGetVariable() to transfer an mxArray from the Matlab Engine to my C++ program. The documentation says: "The limit for the size of data transferred is 2 GB."

Why is this limitation present? Is there a way to increase it without breaking up the array into separate arrays? I need to transfer large arrays, potentially up to 256GB.

I like the simplicity of the Matlab Engine, so I would rather avoid having to break up, transfer, and re-assemble large arrays.

3 个评论
显示 1更早的评论隐藏 1更早的评论

Jonathan 2015-11-11

编辑：Jonathan 2015-11-11

Yes, actually I wrote one. The problem is that I need to solve a large, sparse matrix (Ax=b) on a cluster using Intel Pardiso Sparse Cluster solver.

Unfortunately, integrating Matlab with MPI and using the unsupported Intel MPI compilers seems infeasible.

In fact, I wrote a test program that solves the equation without the MPI code, compiled with GCC using MEX, and it exhibits some weird memory behavior.

So from the above, I concluded that getting this to work in MEX, running on a cluster with MPI is perhaps infeasible.

With regard to transferring 256GB, since I am doing the transfer within a single node (not over a network), it shouldn't take more than a few seconds in theory.

However, are you suggesting that I write my own MEX function that bypasses Matlab Engine to transfer directly using Linux IPC routines?