To use memmapfile you need to have as much memory as the size of the file. The exception to this is if, at the time of mapping, you specify Offset and you specify a Repeat on your Format that implies that only a limited amount of memory is to be used.
I suspect you might find a matFile() more useful for your purpose, provided that you are extracting sub-sections of the array and working with them and writing new sub-sections. If you need to do something like a matrix multiplication of the entire array then you will need to start using swap space, preferably swap space located on an SSD.
With a matrix that large, we would wonder whether perhaps you should be using sparse matrices.
