The following section of the example page provides information on how some of the model's parameters can impact the performance of the model when QAM modulation is used:
There are numerous blocks that perform filtering and correction on the modulated signal so you may need to modify them once you change the modulation schema. The following documentation page provides additional information on the bit error rate and it may give you a better idea of what could be impacting the modified model's BER: