Writing "ymax=x;" doesn't do the job. If 'ymax' is not a scalar value, it needs to be a 'function handle'. The same applies to the integrand function "F=((y^3)/(x^2+y^2));".
Note also that your integration limits are in backwards order. To integrate w.r. y first and then x, you need to write:
a=integral2(F,xmin,xmax,ymin,ymax);
That is the way 'integral2' is defined.
You can read about these requirements at:
http://www.mathworks.com/help/matlab/ref/integral2.html
and function handles at:
http://www.mathworks.com/help/matlab/function-handles.html
