Not sure I follow everything that is going on with your code, but here are my observations:
self.setMessageHandler(@self.defaultMessageHandler);
In the above line of code, it appears you are passing in a function handle, and then using it in your mex routine as follows:
const mxArray* functionHandle = prhs[2];
:
module->setMessageHandlerCallback(const_cast<mxArray*>(functionHandle));
The problem with this approach is that you create the function handle in your m-code on the fly as a function argument. That means that this variable is a temporary variable that gets destroyed immediately after the self.setMessageHandler function call. So the address of this function handle variable is no longer the address of a valid mxArray after this line. But that is the mxArray address you store for downstream use in your callback. When the callback tries to subsequently access this destroyed variable bad things happen. I am actually surprised that you didn't crash MATLAB with a seg fault because of illegal memory access. (Maybe there happened to be another valid MATLAB variable stored at that address at the time of your msIsClass check)
To fix this, you need to ensure that the function handle you are storing for you callback is persistent, and not a variable that can be destroyed in any way at the m-file level. You could make the function handle at the m-file level a persistent variable instead of a temporary variable, but that leaves your mex routine vulnerable to a crash if this variable ever got inadvertently cleared at the m-file level. So maybe the best thing to do is to create a duplicate of the prhs[2], make it persistent inside the mex routine, and then have some clean-up code (e.g., mexAtExit) that will destroy it at the appropriate time (e.g., when the mex routine gets cleared or when another function handle gets passed in etc.). Something like:
const mxArray* functionHandle = mxDuplicateArray(prhs[2]);
mexMakeArrayPersistent(functionHandle);
with other added code for destroying this functionHandle per my above comments.
