I have a feeling that the range of data in B isn't what you think it is. Using the compositing method involving -1 is probably causing problems. If B contains very small negative values near zero, A.*B will result in those regions becoming positive. Then when you set your alpha data to A>=0, those regions show up.
Let me see if I can make an example.
B = repmat(linspace(-0.1,1,256),[256 1]); % the image (numeric)
A = B>0.4 & B<0.6; % the mask (logical)
A = double(A);
A(A==0) = -1; % Change zeros to -1 to keep the zero values in matrix B
C = A.*B;
imagesc(C,'alphadata',C>=0);
caxis([0 1])
colormap jet
Note the extra area selected on the left. It's also possible that the issue isn't negative values in the image or mask, but simply the fact that A==0 will fail to select any near-zero (positive) values in a floating point copy of what should properly be a logical mask.
If you just want to mask off everything but what's defined by the (logical) mask, just use the mask.
B = repmat(linspace(-0.1,1,256),[256 1]); % the image (numeric)
A = B>0.4 & B<0.6; % the mask (logical)
imagesc(B,'alphadata',A)
colormap(jet)
