You have one too many end statements in your function.
In the line
x(j)=(b(j)-a(j,1:j-1)*x(1:j-1)-a(j,j+1:n)*x0(j+1:n))/a(j,j);
when j = 1, a(j,j+1:n) is a 1 x 2 vector, and x0(j+1:n) is a 1 x 2 vector. You have the * operator between two 1 x 2 vectors. The * operator is algebraic matrix multiplication, which requires that for an M x N array "*" a P x Q array, that N and P are the same -- M x N * N x Q giving an M x Q result. 1 x 2 * 1 * 2 does not follow that rule and so is an error. Possibly you want the .* operator instead of * but possibly not.
You also have the problem in that line a(j,1:j-1) is empty, which would make the result of a(j,1:j-1)*x(1:j-1) empty as well. An empty array minus something is empty, so you are down to (b(j) - empty)/a(j,j) but a scalar minus empty is empty too, so you are now down to empty / a(j,j) which will be empty as well, leaving you with x(1) = empty . Which is a problem: you cannot store a 0 x 0 array into a slot that expects 1 value.
