Clarification regarding 2GB transfer limit between MATLAB and Python
3 次查看(过去 30 天)
显示 更早的评论
Could someone please clarify what exactly is meant by 'The size of variables passed between Python and MATLAB is limited to 2 GB when you call a Python function out-of-process. This limit applies to the data plus supporting information passed between the processes.' on https://uk.mathworks.com/help/matlab/matlab_external/limitations-to-python-support.html?
I want to run a sensitivity study using the MATLAB-based UQLab library (https://www.uqlab.com/), which comprises 12 evaluations of a custom Python model spanning many files. The Python model (Python 3.9) is treated as a black box function, which is executed from MATLAB as follows (NOTE: pyenv(ExecutionMode = "OutOfProcess")):
pyrunfile(strcat("python_file.py ", input_variables), output_variable);
where
"python_file.py " is the master script managing my Python model
input_variables is a JSON string of model inputs
output_variable is a single float constituting the model output
Halfway through the 11th model evaluation, MATLAB returns a Python package-specific error ("Python Error: TypeError: Numerical inputs to HAPropsSI must be ints, floats, lists, or 1D numpy arrays."). Now, when I ONLY run the 11th model evaluation I don't get an error, and the results of the individual evaluation are identical to those from the full set of evaluations up to the point where the latter failed. Moreover, when I run all 12 evaluations on a subset of my model (less computationally/memory intensive), I don't get an error either. After repeating the aforementioned steps multiple times, the error message always occurred in the exact same location, halfway through the 11th model evaluation.
All this suggests to me that I am exceeding the 2GB transfer limit between MATLAB and Python, but I don't understand what is meant by 'size of variables passed between Python and MATLAB'? The inputs and outputs described above are only a few byte, but I wouldn't know what falls under 'supporting information'.
0 个评论
回答(1 个)
Sarthak
2024-4-30
Hi Nils,
I think you are facing this error because you are transferring full simulation data for each run to MATLAB which is exceeding the transfer limit. Since you want to run a sensitivity analysis, methods such as Sobol’ indices are usually applied to a limited set of relevant quantities of interest post-processed from your simulation. I believe only this information should be transferred and not the full simulation data for each run which should resolve this error.
I hope this helps!
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Call Python from MATLAB 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!