remove duplicate rows from text file

6 次查看（过去 30 天）
michael
michael 2018-7-19
评论： Walter Roberson 2018-7-20
I'd be glad to have some advise on the fastest way to read a text file which contains duplicate rows, remove all the duplicates and write it back.
Matlab version R14 SP3
  2 个评论
Paolo
Paolo 2018-7-19
What does the text file look like? Could you attach it to your question?
michael
michael 2018-7-20
couple of hundreds of following rows:
F.field1 = ProtoField.uint16("abc", "ABC", base.DEC)

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

Guillaume
Guillaume 2018-7-19
lines = strsplit(fileread(yourtextfile), '\n')); %read file and split into lines
lines = unique(lines, 'stable'); %remove duplicate lines
fid = fopen(newfilename, 'w'); %open file for writing
fwrite(fid, strjoin(lines, '\n'), 'char'); %merge lines and write
fclose(fid);
  4 个评论
显示 2更早的评论
Guillaume
Guillaume 2018-7-19
Matlab version R14 SP3
Wow! You're only 25 versions behind the current version, soon to be 26.
While it's probably fairly easy to write a version of strsplit using regexp that would work with your version and strjoin is probably not too complicated either, you'd also have to write your own unique function since the 'stable' option didn't even exist back then. That's a more complicated task particularly as I'd have to pore over the online documentation to find out what was possible or not back then.
I'm sorry you're on your own there. At some point you've got to let go of legacy code.
Walter Roberson
Walter Roberson 2018-7-20
lines = strsplit(fileread(yourtextfile), '\n')); %read file and split into lines
[ulines, ia] = unique(lines);
lines = ulines(sort(ia));
fid = fopen(newfilename, 'w'); %open file for writing
fprintf(fid, '%s\n', lines{:});
fclose(fid)
Together with
function S = fileread(filename)
fid = fopen(filename, 'rt');
if fid < 0
error('file does not exist');
end
S = fread(fid, [1 inf], '*char');
fclose(fid);
function C = strsplit(S, delim)
C = regexp(S, delim, 'split');

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Data Import and Export 的更多信息

标签

产品


版本

R14SP1

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by