You cannot directly pass two inputs to be optimized to fmincon, and you cannot directly return two outputs from fmincon.
What you should do is create a vector of values, taking either [b, A] or [A, b] and reshaping to a vector. Inside your objective function, extract the appropriate components from the input vector and reshape as needed:
function cost = myobj(x)
b = reshape(x(1:50), [], 1);
A = reshape(x(51:end), 50, 50);
do your stuff
end
Construct your constraints around the same vector configuration.
x0 = reshape([b0, A0], [], 1);
fmincon(@myobj, x0, ....)
I would suggest that in your case, you have a look at Problem Based Optimization, which will automatically do all packing and unpacking for you (but depending on the complexity of your objective and constraints might have some trouble calculating some values.)
