Optimising my data importer for large datasets

Question

HC98 2023-3-26

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1935489-optimising-my-data-importer-for-large-datasets

编辑： Matt J 2023-3-26

So I have this

txtFiles = dir('*.txt') ; %loads txt files
N = length(txtFiles) ; 
Numit = N;
[~, reindex] = sort( str2double( regexp( {txtFiles.name}, '\d+', 'match', 'once' ))); % sorts files
txtFiles = txtFiles(reindex);
for i = 1:N
    data = importdata(txtFiles(i).name);
    x = data(:,1);
    udata(:,i) = data(:,2) ;
end

I have quite a large dataset (well over 200 files) and it takes ages to load things. How can I speed this up? Is there some sort of prepocessing I can do like merge all the files into one or something? I don't know...

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Walter Roberson 2023-3-26

I wonder if using a datastore would be appropriate for your work?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Matt J 2023-3-26

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1935489-optimising-my-data-importer-for-large-datasets#answer_1201259

编辑：Matt J 2023-3-26

在 MATLAB Online 中打开

I don't see any pre-allocation of udata. Also, nothing is being done with x, so it will cut down on time if you don't create it.

udata=cell(1,N);
for i = 1:N
    data = importdata(txtFiles(i).name);
    %x = data(:,1);
    udata{i} = data(:,2) ;
end
udata=cell2mat(udata);

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Matt J 2023-3-26

编辑：Matt J 2023-3-26

在 MATLAB Online 中打开

If the data files have many columns, it will also go faster if you read in only the first two columns, maybe using textscan.

udata=cell(1,N);
for i = 1:N
    fileID = fopen(txtFiles(i).name);
    data = textscan(fileID,'%f %f %*[^\n]');
    fclose(fileID);
    
    udata{i} = data(:,2) ;
    
end
udata=cell2mat(udata);