The behavior you are seeing is a result of Windows trying to "be nice" and allocate the memory you have requested. Since you have a 64-bit address range, it will try as hard as it can to satisfy your request .... although when things are swapped out to disk, they get really slow.
I am not aware of a way to limit an application's memory usage on Windows. I checked Group Policy and found nothing there, I would think that is where you would be able to control such a thing.
A way you could limit this type of behavior is to set your page file size to a certain amount. This way, instead of Windows trying to grow the page file size automatically, it would stop and you would likely get the out of memory error.
