Changing a colon to a 1 means providing plot with a vector for x and a scalar for y (or vice versa). When you do that, one line is created for each element of the vector, and each line has one point. You cannot see a line with one point unless it has a data marker. So either use a data marker for plots like that, or modify your plot call to provide two vectors.
theta = pi/6;
x_mat = rand(1,10);
y_mat = rand(1,10);
r = [cos(theta), sin(theta); -sin(theta), cos(theta)];
b = [x_mat; y_mat];
a = r*b;
plot(a(1,:),a(2,:)) % original
plot(a(1,1),a(2,:),'o') % replacing 1st colon with 1, and using a data marker
plot(a(1,:),a(2,1),'o') % replacing 2nd colon with 1, and using a data marker
plot(a(1,1)*ones(1,size(a,2)),a(2,:)) % plotting 2 vectors
plot(a(1,:),a(2,1)*ones(1,size(a,2))) % plotting 2 vectors
