It appears from the diagram that your p, u, and s are all scalars, 1 x 1. Your k is 3 x 3, so k*s, (3x3)*(1x1) will be 3x3 and negative of that will be 3x3. myvar is u which is 1 x 1 so the -myvar will be 1x1 and 1x1 plus 3x3 will be 3x3 . Your Fx is 3x1 so negative of that is 3x1 . But then you are trying to add that 3x1 to a 3x3 which is going to fail.
Your Ex is 3x3 and inv(3x3) will be 3x3. But why are you taking inv() of the identity matrix? Why are you taking inv() at all? inv() should be avoided due to numeric instability. inv(Ex)*something should be rewritten as Ex\something
Your diagram indicates that you want your y to be 3 x 1. With the 3x3 inverse there being multiplied by something, in order for the result to be 3 x 1, the right hand side would have to be 3 x 1 . So in your (-Fx - myvar - k*sign(s)) expression you are aiming for a 3 x 1 result
If your s were 3 x 1 then k*sign(s), (3x3)*(3x1) would be 3x1 and you would be all set.
It looks to me as if possibly you are expecting your derivative block to output 3 x 1 and your gain to output 3 x 1 and the circle with + + to combine those into 3 x 1, but I suspect that if you check you will find that at least one of those is configured to output 1 x 1 instead of 3 x 1.
