read inconsistent ascii file to matrix

Question

Tom DeLonge 2019-3-27

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/452816-read-inconsistent-ascii-file-to-matrix

评论： Tom DeLonge 2019-4-10

I'd like to obtain maximum performance in reading a file containing both, numeric and non-numeric lines. The files typically look as such:

% comment
text 1.49
1.52 -5.3 8.9710
3.629 -5.77 9
another text and numbers
% comment again
1 2 3
and so on

The file can easily contain 1 million lines.

I would like to obtain two cell arrays:

One that contains all rows that match %f %f %f , i.e. a numeric triplet. Already parsed as numeric doubles. Invalid lines should show up as empty entries or NaN.
Another matrix, that contains all rows that did not match cell-array 1. Still as cellstr, prefereably with trimmed whitespaces.

Obtaining matrix 2 is sort of simple if you already have 1: simply by issuing textscan, and setting all rows that did not match 1 as empty. However, I struggle in obtaining cell array #1. textscan will stop reading once it encounters invalid lines.

In a working example I used sscanf and parsed everything line-by-line. This took about 15s for 1 million lines. Since textscan can read the whole file in less than a second, I am confident that there is room for improvement...

4 个评论
显示 2更早的评论隐藏 2更早的评论

Jan 2019-4-1

编辑：Jan 2019-4-1

What is the meaning of searching for ["a" "e" "i" "o" "u" "A" "E" "I" "O" "U"] ? What do you call "al lot of memory"? Can you provide an example file?

Tom DeLonge 2019-4-9

Sorry, I was on vacation previous week.

I found it to be the fastest way to find all rows that contain also non-numeric data. As said above, it is faster than a regular expression since in my case all text-containing rows do have a vowel in it.

By a lot of memory I mean that the data array will occupy about 100MByte of RAM for a 10MByte text-file (factor of 10 overhead). While 100MByte is not so dramatic yet, for even larger file this will be even worse.

The file is proprietary, which means I cannot provide an example file. But the few lines I've shown above should come pretty close...

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Jan 2019-3-27

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/452816-read-inconsistent-ascii-file-to-matrix#answer_367698

编辑：Jan 2019-4-9

在 MATLAB Online 中打开

Data  = fileread(FileName);
C     = strsplit(Data, char(10));
% [EDITED] Remove comments:
C(strncmp(C, '%', 1)) = [];
match = true(size(C));
NumC  = cell(size(C));
for iC = 1:numel(C)
    % [EDITED2] Small shortcut:
    aC = C{iC};
    if ~isempty(aC) && any(aC(1) == '1234567890-.')
      [Num, n] = sscanf(aC, '%g %g %g');
      if n == 3
          NumC{iC}  = Num;
          match(iC) = false;
      end
   end
end
TextC = C(match);

Is this your current version using a loop? How long does it take?

5 个评论
显示 3更早的评论隐藏 3更早的评论

Jan 2019-4-10

@Tom: textscan is fast for valid inputs. Then I expect fscanf to be even faster. But as soon as the input cannot be caught by a simple format specifier, the processing gets much slower.

Some C code will be faster also, but very tedious to write. It must import the file line by line, but you have to create a buffer, which must be able to contain the longest line also. Unfortunately you do not know the length in advance and the same for the number of outputs. Re-allocation the output array dynamically is a mess in C. So maybe the code runs some seconds faster, but you need a lot of hours for writing and testing. Therefore I like MATLAB.

Tom DeLonge 2019-4-10

Yes, I do agree and understand the limitations of textscan. Thank you for the insights!

请先登录，再进行评论。

Answer 2

Guillaume 2019-3-27

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/452816-read-inconsistent-ascii-file-to-matrix#answer_367697

编辑：Guillaume 2019-3-27

在 MATLAB Online 中打开

Unfortunately, there's no ignore invalid lines for textscan, so you're going to have to parse the file line by line, or implement the parsing in mex.

The following takes about 10s on my machine for a million lines. It's probably similar to what you've done already:

function [num, text] = parsefile(path)
    lines = strsplit(fileread(path), '\n');
    num = cellfun(@(l) sscanf(l, '%f %f %f')', lines, 'UniformOutput', false);
    text = lines(cellfun(@isempty, num));  %could use cellfun('isempty', num) for a marginal speed gain
end