Using a Monte Carlo method to solve an area like this would be a terrible method to use on that problem, either requiring huge numbers of random points or being fraught with relatively large errors.
Furthermore it seems a shame to even use numerical quadrature when the integrals involved have an explicit solution which will yield the answer you quoted, pi*(R^2-a*b), exactly.
But if you are intent on doing it numerically, then it should be set up correctly. If, as it appears in your image, you are doing it using polar coordinates, r and phi, the inner radial limit should have been
a*b/sqrt(b^2*cos(phi)^2+a^2*sin(phi)^2)
I have no idea what you were using as an inner limit here but it is very wrong, and that is not the fault of the Gauss-Legendre integration method. Also it is important that you use the correct Jacobian factor, which with polar coordinates would be just r.
It is not essential that you use Gauss-Legendre integration here. Since the circle and ellipse are entities for which precise values can easily be calculated, there is no great disadvantage in using larger numbers of evenly-spaced points as in trapezoidal integration. Gauss-Legendre is most useful when there is a heavy computation penalty to pay for calculating an integrand and the number of such computations needs to be kept to a minimum.
Also note that it is silly to use numerical integration along the radial lines for the inner integral when you are merely computing the integral of the Jacobian, r. Its indefinite integral is r^2/2 so the definite integral would be the difference between r^2/2 at the outer limit, R, and the inner limit which I gave above.
In computing the resulting outer integral with respect to phi you might need to use numerical methods. However, if you had used cartesian coordinates originally instead of polar coordinates, the whole problem would have been easier and numerical methods would not be needed at all.
