Multi-thread parsing and loading thousands of csv files

Question

George Li 2024-6-12

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2127781-multi-thread-parsing-and-loading-thousands-of-csv-files

评论： George Li 2024-6-12

I have a folder with 2500 csv files, each 15MB each. I currently have a script that reads each csv into a cell array container as follows at the bottom.

Unfortunately this serial process takes a very long time to open each csv one by one.

Ideally I would like to multi-thread or open multiple csv files in parallel and save them into either their own set of cell arrays per 'thread' and later combine and sort them, or into one big cell array as it is currently.

%% IMPORT FILES
directory = '\\headnode\userdata\George\ANSTO\ANSTO Day 2\Data\D14\';
datafiles = dir(append(directory,'*.csv'));
N=length(datafiles);
a = 0;
data = cell(1,N);
f = waitbar(a,'Importing Data...');
for i = 1:N
    data{i} = read_csv(strcat(datafiles(i).folder, '\', datafiles(i).name));
    waitbar(i/N,f);
end
waitbar(1,f);
close(f);

2 个评论
显示无隐藏无

Stephen23 2024-6-12

Alternative: avoid loading them all into memory by using a datastore:

https://www.mathworks.com/help/matlab/large-files-and-big-data.html

https://www.mathworks.com/help/matlab/datastore.html

https://www.mathworks.com/help/matlab/tall-arrays.html

George Li 2024-6-12

Thanks Stephen. I have tried the Datastore method with a filedatastore and using the above code in function form as a custom readall with parallel on function and it is slower than the parfor method quite significantly. Fortunately for me I can fit all data into memory at the current stage

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Ganesh 2024-6-12

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2127781-multi-thread-parsing-and-loading-thousands-of-csv-files#answer_1470766

编辑：Ganesh 2024-6-12

在 MATLAB Online 中打开

Hi @George Li,

You will be able to parallelize the process with a "parfor" instead of using the "for" loop. Using parfor will require a "Parallel Computing Toolbox" license. The implementation would look as follows:

%% IMPORT FILES IN PARALLEL
directory = '\\headnode\userdata\George\ANSTO\ANSTO Day 2\Data\D14\';
datafiles = dir(append(directory,'*.csv'));
N = length(datafiles);
data = cell(1, N);
if isempty(gcp('nocreate'))
    parpool; % Adjust the number of workers as needed, e.g., parpool(4)
end
% Using parfor for parallel processing
parfor i = 1:N
    data{i} = readmatrix(strcat(datafiles(i).folder, '/', datafiles(i).name));
end
% Since waitbar updates are not possible inside parfor, consider alternative progress indication
disp('Data Import Complete');
delete(gcp('nocreate')); % You may choose to delete the parpool

The limiatation to this is that, you will not be able to update the "waitbar" as you are running all it parallely. You might also need to ensure that you have enough RAM to store all the ".csv" files. From your description, the files alone seem to be over 36GBs! The slowdown might also be due to the same reason.

You might want to consider processing the CSVs as a batch.

2 个评论
显示无隐藏无

Sam Marshalik 2024-6-12

Just wanted to mention that you can use DataQueue to still have a waitbar with parfor or parfeval. You can learn more about it here: Send and listen for data between client and workers - MATLAB (mathworks.com). This will let you read in the files in parallel and still maintain an idea of how many files you have read in vs. how many are left.

George Li 2024-6-12

Thank you both. I have tried parfor with a pool of 36 processes and it now takes 178s to finish ingesting all the data vs ~20 mins with the original for loop! Thank you for your help this is perfect.

Fortunately this is short enough now that I don’t really need a wait bar anymore but I will be trying out the data queue method you have posted anyway.

请先登录，再进行评论。