The precedence of .* operator is higher than : operator. You can find more about operator precedence in MATLAB here. So you can get the desired output by using parentheses as shown (1:3:5).*(1:3:5).
1:3:5.*1:3:5 is interpreted as:
1:(3:(5.*1):3):5 -> 1:(3:5:3):5 -> 1:3:5 -> [1,4]
Also note that the .* operator is not being ignored.
