Put in a breakpoint on this line:
R_tow=PH0*(1-tow)*(1-(qfunc(alpha+(sqrt(tow*fs)*Y))))*log2(1+(((tow*Pa/1-tow)-sum(Pi(2:N)))*yi(1))/z)+beta;
and check in the workspace that nothing contains NaNs before you execute this line. If it does then that is the cause and you need to work out what is causing that in whatever component it is.
If you have no NaNs before that line then they are created on that line.
I would suggest breaking that line down into several lines with intermediate variables, at least while you debug. If you prefer to have it all in one line for readability you can put it back that way once it works. But to find the source of problems like this it is a lot easier to look at individual components.
You can also paste the individual bits of it onto command line with a breakpoint if you prefer and examine them that way. I can't remember off the top of my head exactly what type of operations return NaNs, for example division by 0 returns Inf and handing negative numbers to log2 results in a complex result, not a NaN.
