Best route might be to properly differentiate J with respect to delta_Uk and solve the normal equations and find an analytical solution to that system of equations, and check/verify that it is the global minimum - this is after all some kind of 2-D quadratic equation. It should be possible to solve with the symbolic tools available too for general forms of the constant matrices if you define those variables as arrays:
syms F [4 5]
and so on. For a numerical solution you simply have to define YY and J as functions which you can do like this:
F = rand(4,5);
Xf = rand(5,1);
phi = rand(4,2);
Q = rand(4,4);
R_bar = rand(2,2);
Rs = rand(4,1);
YY = @(delta_Uk) F * Xf + phi * delta_Uk;
J = @(dUk) (transpose(Rs - YY(dUk))) * Q * (Rs - YY(dUk)) + (transpose(dUk)) * R_bar * dUk;
This gives you your cost-function J as a function-handle which you can treat pretty much identically like ordinary matlab-function. To minimize it you can simply call fminsearch:
best_dUk = fminsearch(J,[1;2])
Which should give you the optimal value for delta_Uk.
HTH
