This behaviour is expected and correct. I also tried reproducing your code in R2017a and got the same results as R2020a.
The rectangle function plots into the current axes without clearing existing content from the axes. Please refer to the description subsection in the documentation of rectangle function: rectangle documentation.
To open a separate plot every time you run your code you can do this:
close all;
rectangle('Position',rand(1,4));
hold on;
plot(rand(),rand(),'o')
hold off;
