1/m is a scalar/3x3 hence the error. Normally I would advise backslash here, but that brings up another issue. If M, C, and K are all 3x3 matrices, then I would assume x must be a 3x1 vector (at least). So you have a 2nd order equation with a three element vector, making this a 2x3 = 6th order system. Your state vector needs to be 6 elements to handle this, not 2 elements. I.e., the six elements of the 6 element state vector y would be defined as
y(1) = x1
y(2) = x2
y(3) = x3
y(4) = x1'
y(5) = x2'
y(6) = x3'
So your initial conditions must have six states:
And inside your derivative function would be something like this:
y = x(1:3);
ydot = x(4:6);
dxdt = zeros(size(x));
dxdt(1:3) = ydot;
dxdt(4:6) = m \ (F - c*ydot - k*y);
Is it intended that the right hand side, f*sin(wt), is a scalar in this matrix DE?
If x really is supposed to be a scalar, then what are the 3x3 matrices for?