A few observations:
unsigned short *hx;
:
hx = mxGetPr(prhs[0]);
The above code will only work properly if prhs[0] is passed in as a uint16 class variable. You should check for this at the front of your mex routine.
double *link, *x, xsz;
:
link - mxGetPr(prhs[1]);
The above lines will not work at all since you inadvertently used a - instead of a = in the second line. And even after you fix that, be advised that prhs[1] will need to be passed in as a double class variable, so you should check for that. The way you currently have things coded, link is not initialized and gets passed to your unravel function, where it gets dereferenced (one likely cause of the crash). In fact, I would not be surprised that this line in and of itself could cause a crash since the result of the calculation is quite likely an invalid address.
And then this call:
unravel(hx, link, x, xsz, hxsz);
As near as I can tell x is never initialized to anything. So x is not initialized and gets passed to your unravel function, where it gets dereferenced (another likely cause of the crash).
If x is supposed to be an output, then you need to create an appropriate plhs[0] output (class and size) and then do this prior to your unravel call:
plhs[0] = mxCreateWHATEVER(WHATEVER) <-- fill in with appropriate creation routine and arguments
x = mxGetPr(plhs[0]);
