Hello Said,
As per my understanding, you want to tokenize text with special characters and case sensitive letters and want to retain the grouping and case sensitivity of text in the tokenized output.
One of the ways to achieve that is by specifying custom tokens as an argument to "tokenizedDocument" function. The code snippet below shows an example:
str="I am a BeGiNnEr in C# , C++"
documents = tokenizedDocument(str,'CustomTokens',["C++" "C#"])
In the output, it can be observed that the case sensitivity of the characters is retained(this is by default) and the custom tokens specified C# and C++ are not tokenized.
Please refer to the MathWorks documentation on tokenizedDocument to know more about other options available for special characters handling.
