"By default, the units are normalized to the figure. The lower left corner of the figure maps to (0,0) and the upper right corner maps to (1,1). "
What's wrong is precisely what the error message says -- by default annotation uses Normalized units and you have passed something outside the range of [0 1] for either x or y (or both) without changing the 'Units' property to something else
Units — Position units
'normalized' (default) | 'inches' | 'centimeters' | 'characters' | 'points' | 'pixels'
are your choices, axes limits aren't a choice for some reason I've never fathomed...