Hi Chandana Kiran,
I understand that you are looking for some reference implementation of vestigial side band modulation and demodulation in MATLAB.
Here are few reference link that might be useful:
- MATLAB Answer, "Vestigial side band code": https://www.mathworks.com/matlabcentral/answers/486225-vestigal-side-band-matlab-code
- Single sideband amplitude modulation (ssbdemod) function: https://www.mathworks.com/help/comm/ref/ssbdemod.html
Hope this helps!
Regards
Arun