Hi MJ,
Optimization toolbox functions will minimize over "X" where X can be a matrix or vector. Internally, matrices get flattened out into vectors (i.e. x = X(:)). When calling your function, the optimization solvers will call with a matrix of the correct size.
You can pass your constraints in a similar way, although beware about the sizes of the linear constraint matrices. Their sizes should match with the vector length (size(A,2) == numel(X)).
Try it out:
% Define B and initial guess at solution X0
[Xsol,fval] = fminunc(@(X) norm(X-X*B*B','fro'),X0);
