How to split a huge string array efficently

Question

endystrike 2020-7-24

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/569830-how-to-split-a-huge-string-array-efficently

回答： Walter Roberson 2020-7-24

Hi everyone,

I'm trying to split a huge string (~8.5mb, ~11.500 rows x ~400 columns) efficiently, but I cannot do that without a quiet slow "for" loop I cannot remove.

The number of colums may change from a file to another one so it's not possible for me to determin initially a unique format of the file and then import it according to it.

%% getting data from .txt => really fast
tic
disp('importing file');
a = string(textread([pwd '\test.txt'],'%s','headerlines',1)); %#ok<*DTXTRD>
toc
%% splitting each row in colums by delimiter ";" => slow
tic
disp('splitting each row by ";"');
b = strings(length(a),length(strsplit(a(1),';')));
for k=1:length(a)
    b(k,:) = strsplit(a(k),';');
end
toc
%% date(str) to datenum => really fast
tic
disp('conv date to datenum');
dat1 = datenum(b(:,1),'yyyy-mm-dd');
toc
%% str to logical => really fast
tic
disp('converting data to logical array')
dat2 = logical(strcmp(b(:,2:end),'1')); %super fast
%dat2 = str2double(b(:,2:end)); %very slow
toc
% disp('converting data to logical array - 2'); %super fast as well
% tic
% dat2 = zeros(size(b));
% dat2(strcmp(b(:,2:end),'1')) = 1;
% toc

Thanks everyone! :)

Source file sample

3 个评论
显示 1更早的评论隐藏 1更早的评论

endystrike 2020-7-24

在 MATLAB Online 中打开

Thanks Walter, I fixed following your advice! :)

tic
a = readtable([pwd '\test.txt'],'delimiter',';');
dat1 = datenum(string(a{:,1}),'yyyy-mm-dd');
dat2 = logical(strcmp(string(a{:,2:end}),'1'));
toc

endystrike 2020-7-24

If you want to put it as an answer, I'll accept it: you helped me a lot and I fixed the issue! :)

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2020-7-24

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/569830-how-to-split-a-huge-string-array-efficently#answer_470164

在 MATLAB Online 中打开

Why not use readtable() ?

I would also point out that textscan() can process character vectors in which the lines are separated by newlines.

Note: in your release if you use detectImportOptions then it would probably automatically figure out that the first column is a date, and would convert it to datetime format.

It will probably also figure out that the other columns are numeric, in which case strcmp() would not be needed, just

date2 = logical(a{:,2:end});

You might need to use 'HeaderLines', 1, 'ReadVariableNames', false