If I recall correctly, MATLAB keeps a pool of small blocks (I do not recall the block size) that it tries to reuse for small variables. When the pool is exhausted or if the memory is too large for the pool, then MATLAB will use malloc() or equivalent. As per usual, this means that the memory might never be given back to the operating systems when it goes out of use, but MATLAB does appear to sometimes shrink its memory in practice.
If you are running a 32 bit version of MATLAB, then you are not going to be able to create an object larger than 2 Gb, due to limitations in 32 bit architectures.
If you are running a 64 bit version of MATLAB then the limit on the size of objects is determined by how many bits of addressing your processor supports. The Intel x64 architecture in its present incarnations supports 48 bits of physical addressing.
Double precision values are 8 bytes, which is 3 bits. So the x64 architecture would support up to 2^(48-3) = 2^45 such values. That is about 1.37E11 objects.
So yes, if you have a 64 bit architecture, then if you increased your swap space far enough you could do zeros(1e10,1) . You will need about 80 GB for that. The full maximum is 256 TB
