Hi Zaid,
It is indeed possible to do so. You can refer to the following examples that explain how you could use custom modulation scheme for analysing the performance of wireless system:
Indoor MIMO-OFDM Communication Link Using Ray Tracing: https://www.mathworks.com/help/comm/ug/indoor-mimo-ofdm-communication-link-using-ray-tracing.html
This example shows how to perform ray tracing in an indoor environment and use the results to build a channel model for a link level simulation with the MIMO-OFDM technique.
You can modify this example to incorporate your custom modulation scheme and analyze the performance of your wireless system. This way, you can evaluate the performance of your new modulation method in different propagation environments.
When creating a ray tracing propagation model, you can configure the model to find paths with up to a certain number of surface reflections and edge diffractions. This flexibility allows you to tailor the model to your specific needs.
You can also refer to the documentation for Ray Tracing for Wireless Communications: https://www.mathworks.com/help/comm/ug/ray-tracing-for-wireless-communications.html
Hope this helps!