I would not be at all surprised if it was poor starting values that might cause the problem.
syms x y
F1 = -9.135789053E+00-log(x)+166.2509975*x+5.84074229*y+(-166.2509975*x^2)/2-5.84074229*x*y+(-19.52527074*y^2)/2;
F2 =-3.51500942E+00-log(y) +5.84074229*x+19.52527074*y+(-166.2509975*x^2)/2-5.84074229*x*y+(-19.52527074*y^2)/2;
fimplicit(F1,[0,5])
hold on
fimplicit(F2,[0,5])
xlabel x
ylabel y
It is the intersection of the red and blue curves where you will find a solution. There appear to be three such intersections, and a third non-solution happens at x=y=0. but that point is a singularity, not a true solution. The other solutions at x==0 and y==0 are also problematic.
But fsolve should have no real problem in finding the solution as found by @Matt J, as long as you give it reasonable starting values. A problem may be that if you are not careful, is if fsolve ever tries to go outside of the legal search space where x>0 and y>0, then fsolve will fail.
See that even vpasolve fails to find a happy solution, if you allow it to choose its own starting values.
[X,Y] = vpasolve(F1,F2,[x,y])
However, other starting values seem to get to a happy place.
[X,Y] = vpasolve(F1,F2,[x,y],[1 1])
So it was very likely a poor choice of starting values that caused the fsolve failure for @Vikash Sahu.
Can the problem be modified to avoid the issue completely? Well, yes. Replace each of x and y with the squares of two variables.
syms xx yy
G1 = subs(F1,[x,y],[xx^2, yy^2]);
G2 = subs(F2,[x,y],[xx^2, yy^2]);
[XX,YY] = vpasolve(G1,G2,[xx,yy])
X = XX^2
V = YY^2
Now fsolve should be more robust too, even with random starting values, though it still might get trapped in one of the "solutions" at x==0 or y==0.
