You are missing a multiplication operator in between M1^2 and the bracket expression in denominator, see below -
Note that if you are providing beta in degrees, the trignometric functions used will also have to be the ones that use degree as input i.e. sind instead of sin.
beta = 35; %example values
m = 2;
gamma = 1.4;
% v v V v
theta = atand((m^2*sind(beta)-1)/(m^2*(gamma+cosd(2*beta))+2))