The determinant should only be used explicitly to solve an eigenvalue problem for symbolic calculation (for example, when you solve a 2-by-2 problem by hand). In numeric computations, the determinant is not robust and not advised to use.
You can compute the eigenvalues and eigenvectors using the EIG function, [V, D] = eig(K, M). This gives you eigenvalues (diagonal of D) and eigenvectors (columns of V) of this problem. The matrices satisfy
norm(K*V-M*V*D) % == 0 up to round-off error
To get the w value you want, you simply take the square root of the eigenvalues, sqrt(diag(D)).
