From the code provided above, it is my understanding that there is a quadratic constraint involving “x”and “y” in addition to the non-negative constraints.
According to MATLAB documentation, it may not be a recommended practice to use “linprog” as it is better suited for problems involving a linear objective function and at least one linear constraint. A “fmincon”solver may be more appropriate in this case as indicated by the error message.
Please refer to the following documentation link on “Linear on Quadratic Objective with Quadratic Constraints” which contains pointers on how to reformulate the problem.