Your problem is in these lines because S has 6284 elements:
X = det([0 -1/I3 0 0; (H-L^2*I9)/(H) S (2*L^2*I3*I6)/(H*C8) 0; 0 0 S -1/I9; -(2*L*I9)/H 0 -(2*L*I3*I6+H)/(H*C8) S]);
Theta = det([S -1/I3 0 0; (L^2*I6*I9)/(H*C2) S (H-L^2*I9)/H 0; 0 0 0 -1/I9; -(2*L*I6*I9)/(H*C2) 0 -(2*L*I9)/H S]);
F_in = det([0 -1/I3 0 0; (L^2*I6*I9)/(H*C2) S (2*L^2*I3*I6)/(H*C8) 0; 0 0 S -1/I9; -(2*L*I6*I9)/(H*C2) 0 -(2*L*I3*I6+H)/(H*C8) S]);
I suspect the following is what you need:
I6 = 3;
L = 0.5;
I3 = 1.2 * I6;
f_n = 1;
k = I3*(2*pi()*f_n)^2;
C2 = 1/k;
C8 = -9.81;
I9 = (I6*L^2)/12;
H = L^2*I3*I9+L^2*I6*I9-4*I3*I6;
omega = [0:0.001:2*pi()];
S = i*omega;
X=nan(size(S));Theta=nan(size(S)); F_in=nan(size(S));
for c=1:length(S)
X(c) = det([0 -1/I3 0 0; (H-L^2*I9)/(H) S(c) (2*L^2*I3*I6)/(H*C8) 0; 0 0 S(c) -1/I9; -(2*L*I9)/H 0 -(2*L*I3*I6+H)/(H*C8) S(c)]);
Theta(c) = det([S(c) -1/I3 0 0; (L^2*I6*I9)/(H*C2) S(c) (H-L^2*I9)/H 0; 0 0 0 -1/I9; -(2*L*I6*I9)/(H*C2) 0 -(2*L*I9)/H S(c)]);
F_in(c) = det([0 -1/I3 0 0; (L^2*I6*I9)/(H*C2) S(c) (2*L^2*I3*I6)/(H*C8) 0; 0 0 S(c) -1/I9; -(2*L*I6*I9)/(H*C2) 0 -(2*L*I3*I6+H)/(H*C8) S(c)]);
end
ANS1 = abs(X./F_in)
ANS2 = abs(Theta./F_in)
