This also happens for imgaussfilt() and others (e.g. using MIMT fkgen() and imfilter()). If you take a closer look, you'll notice it's not strictly integer-values that are the problem areas. It's integers and half-integers.
These defects are an artifact of the kernel size calculation, which must be integer-valued.
bsig = linspace(.9,5.1,1000);
bsize = 2*ceil(2*bsig) + 1;
The significance of these sudden changes can be reduced by explicitly specifying a larger filter size.
bsize = 2*ceil(4*X(i)) + 1;
B=imgaussfilt(A,X(i),'filtersize',bsize);
I tend to prefer a bit more filter support than the default provides, but as always, the defaults are a compromise between quality and speed.