If you were to evaluate f(g(x),b), then MATLAB would first calculate y = g(x) and then evaluate f(y,b). Your main problem here is that MATLAB is trying to evaluate your integrand before calling the integral function. In order to pass a function into another function, it needs to be a function handle.
Furthermore, MATLAB does not support multiplication by juxtaposition alone. You must supply the multiplication operator explicitly.
Finally, the numerical integration function requires you to formulate your function in a way that allows it to be called with an array of inputs, and when given an array of inputs, the function must perform the calculations "element-wise", returning an array of outputs. Consequently, let us take your input and fix it.
You entered
integral((sqrt(9x/10-x^2+xsqrt(21x/5+81/100)))(sqrt(1+((1/(2sqrt(9x/10-x^2+xsqrt(21x/5+81/100)))(9/10-2x+sqrt(21x/5+81/100)+21x/(10sqrt(21x/5+81/100))))^2))),0,6)
OK, so we need to define a function and pass a function handle in. There are two ways to do that. One is to define a function in a MATLAB file. The other way is to define an anonymous function. The anonymous function is the easiest way to go in this case. If it is a function of one variable, x, we will pre-pend @(x) to the expression. This tells MATLAB that what follows will be considered a function of x. It will not be evaluated at that time, but any variables other than x will take their values from the current workspace. They will be "snapshots" of the current values. That's not relevant in this case, but it is good to know for when you create functions with parameters.
OK, so let's define a function and store it in a variable f. At the same time, let's add multiplcation operators where juxtaposition means multiplication. Note that the requirement that the function accept arrays and work "element-wise" means that we should use the operators .* (element-wise multiplication), ./ (element-wise division), and .^ (element-wise power) rather than * (matrix multiplication), / (matrix division, i.e. linear system solving and least-squares), and ^ (matrix powers). We can relax this were scalar multiplication is concerned, since, for example, scalar multiplication 9*x is the same as element-wise multiplication 9.*x. I'll also take the liberty of converting things like 9/10 to 0.9.
f = @(x)(sqrt(0.9*x - x.^2 + x.*sqrt(4.2*x + 0.81))).*(sqrt(1 + ((1./(2*sqrt(0.9*x - x.^2 + x.*sqrt(4.2*x + 0.81))).*(0.9 - 2*x + sqrt(4.2*x + 0.81) + 21*x./(10*sqrt(4.2*x + 0.81)))).^2)))
That should work, but I hope I didn't make a mistake. A better approach, I think, would be to build this up from some shared expressions.
>> g1 = @(x)sqrt(4.2*x + 0.81)
g1 =
@(x)sqrt(4.2*x+0.81)
>> g2 = @(x)sqrt(0.9*x - x.^2 + x.*g1(x))
g2 =
@(x)sqrt(0.9*x-x.^2+x.*g1(x))
>> f = @(x)g2(x).*sqrt(1 + ((1./(2*g2(x))).*(0.9 - 2*x + g1(x) + 21*x./(10*g1(x)))).^2)
f =
@(x)g2(x).*sqrt(1+((1./(2*g2(x))).*(0.9-2*x+g1(x)+21*x./(10*g1(x)))).^2)
>> integral(f,0,6)
ans =
11.700305459021003
