If you output jac, you will notice that the first and third column approach each other quite rapidly.
I didn't examine why this happens, but it makes the Jacobian singular and Newton's method fail.
You might want to examine your (rewritten) problem with MATLAB's "fsolve" first:
format long
fun = @(x,y,z)[x*sqrt(2*y)+2.88^10*z;(z+x)-1/2*(x+0.5*y+z);x+y+z-1];
x0 = [1;1;1/2];
options = optimset('MaxFunEvals',10000,'MaxIter',10000);
sol = fsolve(@(x)fun(x(1),x(2),x(3)),x0,options)
fun(sol(1),sol(2),sol(3))
