Here's my take on your problem.
I've made up my own numbers, and for that reason it doesn't solve very well, and there are a few possible reasons for this.
I've followed your partitioning suggestion with x defined as [lamda3; w3].
The objective function is straight forward, but the nonlinear constraint less so. Below I've used equality constraints, but I suspect that they are too difficult to exactly solve, so a better approach would be to minimise their residuals.
function rc = foptprob()
A = randn(11);
[W,~] = qr(A); % W is orthogonal
omega12 = [1,2]';
lam0 = 6;
target = 105;
W12 = W(:,[1,2]);
x0 = [lam0; W(:,end)]; % start guess
% j = fobj(x0, target, W12, omega12)
% Now try
A = []; B = []; Aeq = []; Beq = []; LB = []; UB = [];
xstar = fmincon(@(x) fobj(x, target, W12, omega12), ...
x0,A,B,Aeq,Beq,LB,UB, @(x) nonlcon(x, W12))
rc = xstar;
end
function j = fobj(x, target, W12, omega12)
lam = x(1);
w3 = x(2:end);
W = [W12, w3];
Omega = diag([omega12(:); lam]);
j = target - trace(W*Omega*W');
end
function [C, Ceq] = nonlcon(x,W12)
% Satisfy the 3 equality constraints
w3 = x(2:end,1);
w1 = W12(:,1); w2 = W12(:,2);
Ceq = [w1'*w3; w2'*w3; w3'*w3 - 1];
C = [];
end
