I am guessing that when augmentation is on, it trains "exactly" the way it trained when it was turned off, but performs a random transformation (rotation, in your case) on each training example and presents that to the network instead of the original training example. The network rarely sees the exact same training example twice, as this is improbable given that the transformations are random, and it rarely sees exact copies of your actual training examples, since the random generator needs to rotate the image (in yours case) by an amount close enough to 0 degrees that it effectively doesn't rotate the image at all when discretization is accounted for, which is possible but improbable.
This is clearly different from what you were expecting; i.e., you expected it to generate one larger set of augmented training data, then break that up into mini batches that it presents to the network over and over again each Epoch, so that more iterations per Epoch would occur and it would see the same images each Epoch.
Unfortunately, this is not addressed in the R2017b documentation at least, and I doubt it is addressed in the 2018a as well. Your question is valid, and Mathworks should probably put more resources on their NN Toolbox for documentation and functionality if they want to be long-term players here.
