Some problems:
1)
/* coppy array and set the output pointer to it */
plhs[0] = prhs[2];
The above line does not copy the input array to the output array. It only copies a pointer to a pointer. Furthermore, you cannot copy input prhs[] pointers to output plhs[] pointers directly. This violates the rules and can cause a MATLAB crash. Probably what you want instead is a deep copy:
plhs[0] = mxDuplicateArray(prhs[2]);
2)
out[s+M1,n+M1].real = out[s+M1,n+M1].real + etc.
The above line does not do the indexing like you think it is doing. This is not the way to index into a 2D array in C/C++. What you are actually doing in the above line is evaluating the result of a comma operator and using that as a single index. Since the result of a comma operator is the value of the 2nd expression, the above line is the equivalent of
out[n+M1].real = out[n+M1].real + etc.
You will need to manually calculate the equivalent linear index corresponding to the 2D element you are trying to update, and use that linear index instead of what you are doing.
3)
int N = prhs[3];
The above line copies an mxArray * type into an integer. This does not get the value into N. To do that you need to do this instead:
int N = (int) mxGetScalar(prhs[3]);
There are probably other errors too, but I can't read your screen shot above. Please delete the screen shot and post the error messages as text instead.
