I assumed that a and b are 3 elements-vector and then solved the equation beta analytically.
a = [1 2 3];
b = [2 3 4];
beta = roots([b(1)^2+b(2)^2+b(3)^2 , 2*(a(1)*b(1)+a(2)*b(2)+a(3)*b(3)),a(1)^2+a(2)^2+a(3)^2-1])
the result is two solution corresponding to two different value of beta.
to verify our solution, simply try it;
norm(a+beta(1)*b)
or :
norm(a+beta(2)*b)
