your cost_function doesn't return a scalar. it should return one number for every input.
it seems that x is a 1xn vector. and u is a vector with same size as x. (for every t i guess!) but in last line of cost_function you write F = x - 1/2*(u*u') . the second term is a scalar beacuse it is inner product of u with itself make this term become 
. but x is still a vector so the result (F) become 1xn vector. F should be integration if x and u are vector with same size and space. you should sum it over time. (with dt !) : 
becoming :
. but x is still a vector so the result (F) become 1xn vector. F should be integration if x and u are vector with same size and space. you should sum it over time. (with dt !) : becoming :F = sum(x + 0.5*u.^2)
also this might not be a good choice becuse it doen't contain any information about t and the integration is over t. if grid of t is uniform and the intervals are equal summation and numerical integration is different in just a constant coefficient which doesn't change the problem. but integration over t might be better soltion in general:
F = trapz(x+0.5*(u.^2)); %numerical integration with Trapezoidal method
if you have the value of t : (t as vector)
F = trapz(t,x+0.5*(u.^2));
