Try, doing following things to see if it works:
1. Turn off the Workspace I/O (to do that, in the Simulink model go to Simulation=>Parameters=> Data Import/Export and uncheck the time and output boxes).
2. Increase the step size of the simulation
3. Reduce the length of the simulation
4. Increase the virtual memory of your computer (system swap space).
If the error still exists, share the exact error or if possible share the model which generates the error.
Also, check if your system is really out of memory by running task manager.
