How to do one hot encoding of unusual letters in matlab?

Question

Sanjana Sankar 2019-7-19

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/472466-how-to-do-one-hot-encoding-of-unusual-letters-in-matlab

评论： Sanjana Sankar 2019-7-31

I have a table of german characters. I would like to one hot encode them so that i can input it to a neural network. How should i go about?

The problem is some of the characters are not accepted by matlab as characters. For example, 'ä' 'ö' 'ü' 'ß'

Regardless, I would like to know how to one hot encode any character from a TABLE in matlab.

Thanks in advance!

3 个评论
显示 1更早的评论隐藏 1更早的评论

Mario Chiappelli 2019-7-19

Was ist heiss Codierung?

Sanjana Sankar 2019-7-22

Please check the link provided by Walter about one-hot encoding

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2019-7-19

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/472466-how-to-do-one-hot-encoding-of-unusual-letters-in-matlab#answer_384002

编辑：Walter Roberson 2019-7-19

https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/ describes One-Hot Encoding (a term I was not aware of)

In MATLAB, see https://www.mathworks.com/help/deeplearning/ref/vec2ind.html

You might want to first construct a list of permitted characters, and map the input into an offset in that list. That will potentially save you from wasting bits on characters such as œ that you are not using.

5 个评论
显示 3更早的评论隐藏 3更早的评论

Walter Roberson 2019-7-30

在 MATLAB Online 中打开

permitted = ['A' : 'Z', 'a' : 'z', 'ä', 'ö', 'ü', 'ß', ' ', '.' ] ;
 [found, idx] = ismember(YourText, permitted);
  assert(all(found), 'unpermitted character detected')
  OneHot = ind2vec(idx) ;

Sanjana Sankar 2019-7-31

OneHot = full(ind2vec(idx,max(idx)));

This is exactly what I was looking for and I got it from your code. Thank You! This works :)

请先登录，再进行评论。

Answer 2

Guillaume 2019-7-30

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/472466-how-to-do-one-hot-encoding-of-unusual-letters-in-matlab#answer_385398

在 MATLAB Online 中打开

file containing the characters that I want to one hot encode. Do you know how I can go about now

%phoneme_set: A cell array of phonemes to one-hot encode
assert(numel(phoneme_set) > 64, 'Cannot one-hot encode more than 64 phonemes with a 64-bit integer')
phoneme_set(:, 2) = num2cell(2 .^ uint64(0:size(phoneme_set, 1)-1))

3 个评论
显示 1更早的评论隐藏 1更早的评论

Guillaume 2019-7-30

在 MATLAB Online 中打开

number                     binary pattern (64 bits)
2^0    0000000000000000000000000000000000000000000000000000000000000001
2^1    0000000000000000000000000000000000000000000000000000000000000010
2^2    0000000000000000000000000000000000000000000000000000000000000100
...
2^63   1000000000000000000000000000000000000000000000000000000000000000

This is what one-hot encoding is. As I commented, this is useful for FPGAs and similar which operate at the bit level. On the generic processor of a computer, it's a complete waste of space but my answer does what you asked.

You could encode the one-hot encoded numbers that I generate as a vector of 0 and 1 (double) for even more waste of space:

phoneme_set(:, 3) = num2cell(fliplr(eye(size(phoneme_set, 1), 64)), 2)

Note that the binary pattern of the 0s in that encoding is 0000000000000000000000000000000000000000000000000000000000000000b and of the 1s is 0011111111110000000000000000000000000000000000000000000000000000b.

However, I suspect that what we have here is a XY problem. You have some unspecified problem doing something and you think you can solve it by using one-hot encoding (without really understanding what it means) so ask about one-hot encoding instead of your actual problem.

Sanjana Sankar 2019-7-30

I have to feed characters to a neural network. And the most common method is one hot encoding. But I haven't found any examples on how it is done in MATLAB.

请先登录，再进行评论。