ode45 returns a y with as many columns as there are values in the initialization vector. Your initialization has two elements so two columns will be returned. You do subtraction of an array of equal, getting an N x 2 array. You square the elements, getting an N x 2 array. You then sum(); by default sum() works along the first dimension, so your output is going to be 1 x 2 . You return that back to the internals of fminsearch, but fminsearch is expecting a scalar.
I speculate that you want another sum() around the sum() so that you get a scalar output.
Note: if you have the System Identification toolbox then if I understand correctly you can use some of the functions there to achieve the purpose you are coding for here.
