Handling out-of-vocabulary word in word embedding

Question

Ismat Mohd Sulaiman 2021-7-5

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/872188-handling-out-of-vocabulary-word-in-word-embedding

评论： Ismat Mohd Sulaiman 2021-8-16

I'm using FastText and my own word embedding on a set of documents. It is being used to detect abbreviations (Y/N) for each word token.

When testing, words that does not have vectors (out-of-vocabulary - OOV words), and discarded and not included in the performance measures (precision, recall, etc.) giving a false result. How do you handle this?

Would you replace all words with NaN values be included in the performance measure? Can the NaN values be replaced with a vector? How would you decide which vector?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Prince Kumar 2021-8-16

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/872188-handling-out-of-vocabulary-word-in-word-embedding#answer_767977

From my understanding your want to handle OOV(out-of-vocabulary) words for your abbreviations detection task. For now MATLAB fastTextWordEmbedding does not handle OOV words.

There are many ways to do it, following are the two popular ones:

Use context around the OOV word. You can use the word embedding of the previous and next word to your current OOV word.
Use synonyms or similar word to get the word embedding for your OOV word.