In a matrix multiplication the length of the row of the first matrix must equal the length of the column of the second one.
In you case transposing the 2nd matrix is enough already:
A = [0 0 0];
B = [1 0 0];
C = [0 1 0];
D = [0 0 1];
E = [0 1 1];
F = [1 0 1];
G = [1 1 0];
H = [1 1 1];
P = [A;B;F;H;G;C;A;D;E;H;F;D;E;C;G;B];
alfa = 45 * pi / 180;
Ry = [cos(alfa),0,sin(alfa);...
0,1,0;...
-sin(alfa),0,cos(alfa)];
G = Ry * P.';
plot3(G(1, :), G(2, :), G(3, :)); % Not G(:, 1)
axis equal
