How can I delete numeric Headings/Delimiters from a large text file

2 次查看（过去 30 天）
Tate Shorthill
Tate Shorthill 2018-1-19
评论： Greg 2018-1-23
I have packets of data each (47,4) (rows,columns) in a massive text file. Each of these packets are separated by a row of numerical headings (1,3). I would like create a script to find all the 3-column rows and delete them thus giving me one massive 4-column data file. (note, all values are numeric) Thanks for the help!
I've attached an example file.

请先登录，再回答此问题。

采纳的回答

Jan
Jan 2018-1-19
编辑：Jan 2018-1-23
inFID = fopen(FileName, 'r');
outFID = fopen([FileName, '.fixed'], 'W'); % [EDITED: 'w' => 'W']
Delimiter = ','; % Or whatever it is
Break = char(10);
while ~feof(inFID)
S = fgets(inFID);
if length(findstr(S, Delimiter)) > 2
fwrite(outFID, S, 'char');
fwrite(outFID, Break, 'char'); % Linebreak
end
end
fclose(inFID);
fclose(outFID);
Import each line. Export it only, if it is recognized as not belonging to the "numerical headings (1,3)". Maybe length(findstr(S, Delimiter)) > 2 is not optimal, so please post a short example of how the kind of lines can be distinguished.
By the way: "Massive" text files are nonsense. Text files are useful only, if they are read and edited by human, but this is impossible for huge data. Think of converting the data to a binary format, if you want to store a matrix only. This would be more efficient.
[EDITED 2] Do the lines to be kept start with a space? Then this might be faster:
S = fileread(FileName);
C = strsplit(S, char(10));
C = C(strncmp(C, ' ', 1));
fid = fopen([FileName, '.fixed'], 'W');
fprintf(fid, '%s\n', C{:});
fclose(fid)
  3 个评论
显示 1更早的评论
Jan
Jan 2018-1-22
I do not see a need to advertise another thread here. My answer does not concern textscan or regexp. But if you have a good reason to do this, post a link. It is not convenient to let the readers search.
Jan
Jan 2018-1-23
If have changed the output buffering by using fopen('W'). See [EDITED 2] for another idea.

请先登录，再进行评论。

更多回答（2 个）

Greg
Greg 2018-1-20
f = 'examplefile.txt';
fmt = ['%*f%*f%*f',repmat('%f%f%f%f',1,47)];
fid = fopen(f,'rt');
data = textscan(fid,fmt,Inf,'Delimiter',{' ','\n'}, ...
'MultipleDelimsAsOne',true,'CollectOutput',true);
fclose(fid);
data = data{1}';
data = reshape(data,4,[])';
Jan's solution is the brute force, won't break if a block has 46 lines instead of 47. Mine is the take advantage of the given repetition. I'll leave it to you to output the data; as with other comments, I don't recommend writing it out to another .txt file. Chances are that file will just need to be read in again - a .mat file or a standard fwrite(...,'double') format would be drastically more appropriate.
  2 个评论
Tate Shorthill
Tate Shorthill 2018-1-23
thanks for your feedback. I likely will come back to this because the code provided by Jan, while effective, is fairly slow.
Greg
Greg 2018-1-23
You're dealing with ASCII text, slow is kind of your only option. This should be a bit faster than Jan's but keep in mind that his is reading and writing mine is reading only.

请先登录，再进行评论。

per isakson
per isakson 2018-1-20
编辑：per isakson 2018-1-20
And with regular expressions
>> out = cssm('examplefile.txt');
>> out(1:32)
ans =
1 5 0 0
2 15 0 0
3 25 0
>>
where cssm
function out = cssm( ffs )
xpr = [ ... Match the "numerical headings"
'(?m) ' ... "^" and "$" match begining and end of line
'^\x20* ' ... beginning of line and optional space, '\x20'
'[\d.]+ ' ... one or more digits and periods
'[,\x20]+ ' ... list delimiter; one or more comma and space
'[\d.]+ ' ... one or more digits and periods
'[,\x20]+ ' ... list delimiter; one or more comma and space
'[\d.]+ ' ... one or more digits and periods
'\x20* ' ... optional trailing spaces
'\r?\n ' ... zero or one CR and one LF; new line
];
xpr( isspace( xpr ) ) = []; % remove space
str = fileread( ffs ); % read entire file as string
out = regexprep( str, xpr, '' ); % replace "numerical headings" by empty
fid = fopen( 'out.txt', 'w' );
[~] = fwrite( fid, out, 'char' ); % write the modified string
fclose( fid );
end

类别

在 Help Center 和 File Exchange 中查找有关 Text Data Preparation 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by