READ THE EROR MESSAGE. This is important, since the error message tells you whatit thinks is wrong.
Where does it say the error came from?
Error using gcd (line 49)
Double input is out of integer type range.
Error in decrypt_modulus (line 15)
g=gcd(share(i,j),m);
Error in nsharestest (line 39)
dred{i}=decrypt_modulus(red{i});
Starting at the top, it tells you the error came from GCD. GCD is not happy with something. The problem happened inside decrypt_modulus. But the error was in the call:
g=gcd(share(i,j),m);
Are both of those values the proper class?
The point is, red{i} is NOT the issue, at least not directly.
Learn to use the error messages. Read them.
Next, you will find it useful to learn to use the debugger.
help dbstop
A nice trick is to turn the debugger into a mode where it will stop as soon as it sees an error. Then you can check to see what caused the error. Use this call to set the debugger in operation.
dbstop if error