If you're just looking for a best fit plane (for some definition of 'best'), you can use the backslash operator, just like finding the the least-squares fit for a line.
For a line, you have data x and y, but unknown coeffcients m abnd b:
y1 = m*x1 + b;
y2 = m*x2 + b;
y3 = m*x3 + b;
...
This represents an overdetermined system X*M = Y, where we can solve M = [b; m]
X = [ ...
1, x1; ...
1, x2; ...
1, x3; ...
];
Y = [y1; y2; y3;];
M = X\Y;
For a plane, we have the general equation
0 = a*(x - x0) + b*(y - y0) + c*(z - z0)
z = (1/c)*(a*x0 + b*y0 + c*z0) - ((a/c)*x + *b/c)*y);
Let C = (1/c)*(a*x0 + b*y0 + c*z0), A = -a/c, B = -b/c, then for M = [C, A, B], you again have the overdetermined equation Z = X*M, which can be solved via backslash:
X = [ ...
1, x1, y1; ...
1, x2, y2; ...
1, x3, y3; ...
];
Z = [z1; z2; z3;];
M = X\Z;
In general, if you have vectors of data x(:), y(:), z(:) of length n, then you can create the data variables as X = [ones(n, 1), x(:), y(:)]. Below is a full example:
m = 188;
n = 215;
[y, x] = meshgrid(1:n, 1:m);
z = sind(10*x + 5*y) + log(x.*y); % filler data; this would be your A matrix
X = [ones(m*n, 1), x(:), y(:)];
M = X\z(:);
figure;
surf(x, y, z, 'EdgeColor', 'none'); % original wavy data
hold all;
surf(x, y, reshape(X*M, m, n), 'EdgeColor', 'none', 'FaceColor', 'b'); % blue fitted-plane
Alternatively, if you just want to get rid of the peaks, you might consider filtering your data (e.g. filter2).
Hope this helps.
--Andy
