Your ymax function returns a sym result. If you specify the limits as function handles in your call to integral2 those function handles must return a numeric result, not a symbolic result.
If you want to integrate with a symbolic limit, work solely with symbolic variables and call the int function twice instead of calling integral2.
If you want to be able to specify b when you call integral2 rather than when you define the ymax function:
ymax = @(x, b) b.*sqrt(x);
Now when you call integral2 (I'm assuming you've defined fun, xmin, xmax, and ymin before this call) specify b:
% Use b = 1
Q = integral2(fun, xmin, xmax, ymin, @(x) ymax(x, 1));
% Use b = x.^2
Q = integral2(fun, xmin, xmax, ymin, @(x) ymax(x, x.^2));
