Hi Muhammad,
I understand that you are looking for examples related to ray tracing models where you could use custom modulation scheme for analysing the performance of wireless system.
I believe following documentation example link would be useful:
You could also refer following documentation link for more details about ray tracing for wireless communication.