Supply
F=[X,Y;Y,Z]*[Tx;x(1);Txy;x(2);x(3);x(4)]-[x(5);Cy;x(6);x(2);x(3);x(4)]
for MATLAB's "lsqnonlin" if e.g. Cx, Cxy, Vx, Ty, Vy, Vxy are unknown.
Or reorder your equations such that it reads A*x=b with b known and x unknown and solve for x using x=A\b.
Best wishes
Torsten.