Hi Faris,
You are using the support package in two different ways in your description above.
In the second case, you are sending data from your host PC to the board, which is passed to the FPGA, runs through your QAM transmitter and is transmitted. In this case, Simulink is not running in real time and will send data as fast as it can to the board, which is probably fast enough to feed your transmitter.
When you use the blocks generated by the HW/SW co-design workflow, and run the model in External Mode, as you do in your first case, you are generating code from your Simulink model to run on the ARM processor. In this case the ARM will be running according to a schedule configured from your model settings, including the absolute rates in the model. Please see the System Timing section in the documentation. Essentially, you will need to set up your Simulink model so that it is feeding enough data to the transmitter. I suspect if you monitor the underflow port you will see that you are experiencing underflows. There is some more documentation on configuring your software interface model here.
Hopefully this helps.
Neil