The error occurs when the Java that comes with the MATLAB installer suffers a critical memory leak. In order to install MATLAB, try obtaining new installation files in case of a corrupt JRE in the installer.
If the error occurs at MATLAB startup, then the installation may be corrupt. To resolve this, re-download the products again and perform a new installtion.
I hope this will help you.