F=@(x,y)[eqn3;eqn4;eqn5];
Okay, F takes two arguments and ignores them and returns an array of items based on symbolic x and y and possibly other things (we cannot see the code for eqn4 or eqn5.)
fun=@(u) F(u(1),u(2),u(3));
fun takes a vector of 3 or more items and passes the first 3 one by one to the function F that is only expecting two parameters.
Let me emphasize that the parameters you pass to F are going to be ignored. When you construct the array for fun then the values that are going to be captured in the anonymous function will be the x and y that are symbolic variables. if you want the x and y padded in to be used in the expression then subs() them into the vector of equations.
But you are passing the function to fsolve. fsolve does not expect symbolic results. It also cannot proceed properly with == expressions
You should either use vpasolve or else you should leave out the ==0 should use matlabFunction() to construct numeric functions to fsolve
