Without seeing your ODE solver call, I cannot determine what the problem is.
This runs without error for me (in R2020b):
function dx = eq62(t,x)
u = -(2*x(2)-8*x(1)+6*x(3)+8*x(1)*x(3)-(x(3)-2)*(x(1)-x(2)+x(3))+x(1)*(x(1)-1))/(x(1)+1);
dx = [-x(1)+x(2)-x(3);-x(1)*x(3)-x(2)+u;-x(1)+u];
end
[t,x] = ode45(@eq62, [0 5], [0 0 1]);
figure
plot(t,x)
grid
.
