你调用方式不太对，我修改了一下你的代码，但是有warning，选择合适的算法求吧:)
obj.M
function [n,x1,x2,x3] = obj(t)
%UNTITLa(5)a(4)4 此处显示有关此函数的摘要
% 此处显示详细说明
x=t(1);
y=t(2);
a1=t(3);
a2=t(4);
a3=t(5);
d=t(6);
e=t(7);
f=t(8);
[X,Y]= meshgrid(x,y);
z=X.^2+Y.^2;
T=[1,0,0,0
0,1,0,0
0,0,1,0
a1,a2,a3,1];
R1=[1,0,0,0
0,cos(d),sin(d),0
0,(-1)*sin(d),cos(d),0
0,0,0,1];
R2=[cos(e),0,(-1)*sin(e),0
0,1,0,0
sin(e),0,cos(e),0
0,0,0,1];
R3=[cos(f),sin(f),0,0
(-1)*sin(f),cos(f),0,0
0,0,1,0
0,0,0,1];
w=[x,y,z,1]*R3*R2*R1*T;
x1=w(:,1);
x2=w(:,2);
x3=w(:,3);
n=x3-x1^2-x2^2;
Optimization running.
Warning: Trust-region-reflective algorithm requires at least as many equations as variables; using Levenberg-Marquardt algorithm instead.
Objective function value: 7.888609052210118E-31
Local minimum found.
Optimization completed because the size of the gradient is less than
the default value of the function tolerance.
