How to read a UTF-8 encoded text file as a single character vector including white spaces and unicode special characters?

73 次查看（过去 30 天）
Deepu George Kurian
编辑： Rik 2021-2-19
I am trying to read a UTF-8 encoded .txt file, "data.txt" containing sample info like this.
<title>
Fate/kaleid liner Prisma☆Illya (Fate/Kaleid Liner Prisma Illya) - MyAnimeList.net
</title>
If I try;
data = fileread('data.txt');
Sample read data:
<title>
Fate/kaleid liner Prisma☆Illya (Fate/Kaleid Liner Prisma Illya) - MyAnimeList.net
</title>
I lose the UTF8 encoded special characters. Here, '☆' is misread as '☆'.
If I try;
file = fopen('data.txt','r','n','UTF-8');
data = fscanf(file, '%s');
fclose(file);
Sample read data::
<title>Fate/kaleidlinerPrisma☆Illya(Fate/KaleidLinerPrismaIllya)-MyAnimeList.net</title>
I can retain the unicode characters but loses all the white space characters.
If I try;
file = fopen('data.txt','r','n','UTF-8');
data = textscan(file, '%s');
fclose(file);
Sample read data:
11×1 cell array
{'<title>' }
{'Fate/kaleid' }
{'liner' }
{'Prisma☆Illya' }
{'(Fate/Kaleid' }
......
It's a cell broken up by white spaces, even though it did read all the unicode correctly.
Can you give me possible way to overcome this issue?

请先登录，再回答此问题。

采纳的回答

Walter Roberson
Walter Roberson 2020-5-14
file = fopen('data.txt','r','n','UTF-8');
data = fread(file, [1 inf], '*char');
fclose(file)
  3 个评论
显示 1更早的评论
Walter Roberson
Walter Roberson 2020-5-14
You have a problem: 11577.txt and New.txt are ISO-8896-1 Latin1 encoded, but 14829.txt is UTF-8 encoded.
It is sometimes possible to tell the difference between the two encodings, but there is no provided routine for doing that.
If you were using R2020a or later, then fileread() would be enough: R2020a improved encoding detection and automatic use of encodings.
Deepu George Kurian
Deepu George Kurian 2020-5-15
Ahh....... You just reminded me of a stupid mistake I made while acquiring those crude data. Thanks man. I corrected it and it works perfectly now.

请先登录，再进行评论。

更多回答（2 个）

Rik
Rik 2020-5-14
编辑：Rik 2020-5-14
I wrote the readfile function for this goal. It will result in a cell array, but you can concatenate them back to a long char array if you prefer.
data=cell2mat(readfile('data.txt'));
Note: this removes all newlines. You can replace them with spaces like this:
data=readfile('data.txt');
data(2,:)={' '};
data=data(:)';
data=cell2mat(data);
  6 个评论
显示 4更早的评论
Deepu George Kurian
Deepu George Kurian 2020-5-14
I am sorry. Its working now. Guess I made some mistake, the first time.
But I have encountered another issue with this. The character ° is not read correctly.
I have added a file 'New.txt' which contains this character in the link I shared before.
You can use this code on extracted character vector, to check if output is correct.
title = char(extractBetween(data, '<title>', ' - MyAnime'));
Required Output:
' Gintama°: Aizome Kaori-hen'
Observed Output:
' Gintama�: Aizome Kaori-hen'
Deepu George Kurian
Deepu George Kurian 2020-5-15
Sorry man. As Rik pointed out here, that last error was due to my stupidity while acquiring those crude data. Both your answers work perfectly now. I am choosing his, just because I can have one extra function less there. But thanks a lot

请先登录，再进行评论。

MathWorks Support Team
As of MATLAB R2020a, fileread accomplishes the desired task.
  1 个评论
Rik
Rik 2021-2-19
编辑：Rik 2021-2-19
Ran in:
This is not quite true, as it doesn't work on all Unicode characters:
fid=fopen('foo.txt','w','n','UTF-8');
fprintf(fid,'%s','😀');
fclose(fid);
fid=fopen('foo.txt','rb');fread(fid).',fclose(fid);%display raw bytes
ans = 1×4
240 159 152 128
fileread('foo.txt') % show fileread result
ans = ''

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Characters and Strings 的更多信息

标签

产品


版本

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by