How to write ASCII to byte file?

13 次查看（过去 30 天）
Jan
Jan 2024-10-4
编辑： Stefanie Schwarz 2024-10-21
In MATLB versions until R2021a I could write ASCII values stored in a CHAR array into a string as unsigned char by:
data = char([126, 129]); % Exceeds 7 bit ASCII
file = fullfile(tempdir, 'test.dat');
[fid, msg] = fopen(file, 'w');
assert(fid ~= -1, msg);
fwrite(fid, data); % Write uchar until R2021b
fclose(fid);
This wrote the unsigned chars [126, 129] untill R2021b:
[fid, msg] = fopen(file, 'r');
bytes = fread(fid, [1, inf], 'uint8'); % < R2021b: [126, 129]
Since R2021b the bytes are converted to UTF-8 and the file contains [126, 194, 129], so a specification is required:
fwrite(fid, data, 'uchar'); % < R2024b: [126, 129], R2024b: [126, 194, 129]
In R2024b this writes [126, 194, 129] again. Intuitively I've tried:
fwrite(fid, data, 'uint8'); % R2024b: [126, 194, 129] also
Playing with the encoding type in fopen does not help also. My questions:
  1. Is there a low level method to write UCHARs stored in a CHAR vector using fopen / fwrite without a Unicode conversion?
  2. This change of behaviour breaks a lot of my codes. Is this considered to be useful?
  3. What is the best way to write unsigned bytes without dependency to the platform?
A workaround is the casting to UINT8:
fwrite(fid, uint8(data), 'uint8');
But duplicating data in memory without a reason is a waste of time.

请先登录，再回答此问题。

回答（3 个）

Bruno Luong
Bruno Luong 2024-10-4
编辑：Bruno Luong 2024-10-6
bytes = uint8([126, 129])
data = char(bytes); % Exceeds 7 bit ASCII
Not sure about the encoding standards but this seems to do what you want
file = fullfile(tempdir, 'test.dat');
[fid, msg] = fopen(file, 'w', 'n', "windows-874"); % "ISO-8859-1" is even better see Andres"s comment
assert(fid ~= -1, msg);
fwrite(fid, data); % Write uchar until R2021b
fclose(fid);
[fid, msg] = fopen(file, 'r');
assert(fid ~= -1, msg);
bytes = fread(fid, [1, inf], 'uint8');
fclose(fid);
bytes % [126, 129]
  5 个评论
显示 3更早的评论
Jan
Jan 2024-10-5
@Bruno Luong: You wrote "fopen defaults to using UTF-8 in order to provide interoperability between all platforms". I understand and expand: "all plattforms except older Matlab versions".
'ISO-8859-1' works with older Matlab versions also, since the encoding-type was added to fopen. Therefore this is my prefered solution.
Bruno Luong
Bruno Luong 2024-10-5
Yes and it is less commands to change in the code with fopen than fwrite so less risk of error.
Ascii encoding seems always messy and mysterious to me. For example one have to specify the encoding in writing but not in reading.
Please let us know what causes the change of behavior.

请先登录，再进行评论。

Andres
Andres 2024-10-4
编辑：Andres 2024-10-4
Seemingly you have two options:
Either use the ISO-8859 encoding with fopen (see my comment on Bruno's answer) or use 'ubit8' precision with fwrite:
bytes_write = uint8([0, 126, 129, 150, 215, 255]);
data = char(bytes_write);
file = fullfile(tempdir, 'test.dat');
fid = fopen(file, 'w');
fwrite(fid, data, 'ubit8');
fclose(fid);
fid = fopen(file, 'r');
bytes_read = fread(fid, [1, inf], '*uint8');
fclose(fid);
l = dir(file);
l.bytes % 6
isequal(bytes_write, bytes_read) % true
Both variants did the job in R2024a and R2024b.
  3 个评论
显示 1更早的评论
Jan
Jan 2024-10-5
@Bruno Luong: I've asked the support about this difference now.
@Andres: 'ubit8' works. Setting the encoding type seems to be more intuitively. Thank you.
Stefanie Schwarz
Stefanie Schwarz 2024-10-21
ubitn is meant to write "any" bit length, n. So it should be uint8 == ubit8 as long as the data array is uint8. We will look into this, but the recommendation is to specify the encoding when working with char as pointed out by Bruno.

请先登录，再进行评论。

Stefanie Schwarz
Stefanie Schwarz 2024-10-21
编辑：Stefanie Schwarz 2024-10-21
To write exact bytes, create the data using the uint8 data type and write the bytes with fwrite and the uint8 output type:
data = uint8([126, 129]);
file = fullfile(tempdir, 'test.dat');
[fid, msg] = fopen(file, 'w');
fwrite(fid, data, 'uint8');
fclose(fid);
char is subject to unicode conversion in fwrite. You can write specific byte patterns for a specific encoding, like:
data = char([126, 129]);
file = fullfile(tempdir, 'test.dat');
[fid, msg] = fopen(file, 'w','n','windows-1252');
fwrite(fid, data);
fclose(fid);
In older versions of MATLAB, there was a bug where we weren't doing the correct unicode conversions on char values above the ASCII range. However, the bug wasn't noticed because the default encoding allowed characters in the extended-ascii range [128-255], and so the output was the same for all values in [0-255] range.
We fixed this bug in R2021b and ported the bugfix back as far back as R2019b. Here is the official bug report:
fwrite does not write UTF-8 character data correctly (2443431)
The default output encoding of fopen changed to UTF-8 in R2020a which should have resulted in different output file results, however the bug in fwrite meant that it continued to produce the same file. After fixing the bug, any time the input type is a text type (string/char), it's converted to the output encoding before being written to the file. Most users wouldn't have noticed the bug unless they specified the encoding as UTF-8 (or were on Linux). The bug remained undetected for a long time since, despite the code being wrong, it produced the correct output for the default encoding for extended ASCII range characters.
So using the char datatype to store bytes was never correct, but for many cases, coincidentally produced the expected file output due to the default encoding and then because of the bug in fwrite.

类别

在 Help Center 和 File Exchange 中查找有关 Low-Level File I/O 的更多信息

标签

产品


版本

R2024b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by