MATLAB does not permit you to create new operators.
MATLAB does permit you to redefine many of the existing operators, such as redefining plus() to affect the operator known as + . There are only a small number of operators that cannot be redefined. However, doing this for an existing data class is often risky as it often interferes with existing use of the operator.
Chances are that you should create a function instead of an operator. That function might be a method for a new class you define.
In your situation, provided that you have access to the symbolic toolbox, then I would suggest something like:
MyOp = @(f) -diff(f,y,2) + f.^2;
