Multi-thread parsing and loading thousands of csv files

8 次查看（过去 30 天）
George Li
George Li 2024-6-12
评论： George Li 2024-6-12
I have a folder with 2500 csv files, each 15MB each. I currently have a script that reads each csv into a cell array container as follows at the bottom.
Unfortunately this serial process takes a very long time to open each csv one by one.
Ideally I would like to multi-thread or open multiple csv files in parallel and save them into either their own set of cell arrays per 'thread' and later combine and sort them, or into one big cell array as it is currently.
%% IMPORT FILES
directory = '\\headnode\userdata\George\ANSTO\ANSTO Day 2\Data\D14\';
datafiles = dir(append(directory,'*.csv'));
N=length(datafiles);
a = 0;
data = cell(1,N);
f = waitbar(a,'Importing Data...');
for i = 1:N
data{i} = read_csv(strcat(datafiles(i).folder, '\', datafiles(i).name));
waitbar(i/N,f);
end
waitbar(1,f);
close(f);
  2 个评论
George Li
George Li 2024-6-12
Thanks Stephen. I have tried the Datastore method with a filedatastore and using the above code in function form as a custom readall with parallel on function and it is slower than the parfor method quite significantly. Fortunately for me I can fit all data into memory at the current stage

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Ganesh
Ganesh 2024-6-12
编辑：Ganesh 2024-6-12
Hi @George Li,
You will be able to parallelize the process with a "parfor" instead of using the "for" loop. Using parfor will require a "Parallel Computing Toolbox" license. The implementation would look as follows:
%% IMPORT FILES IN PARALLEL
directory = '\\headnode\userdata\George\ANSTO\ANSTO Day 2\Data\D14\';
datafiles = dir(append(directory,'*.csv'));
N = length(datafiles);
data = cell(1, N);
if isempty(gcp('nocreate'))
parpool; % Adjust the number of workers as needed, e.g., parpool(4)
end
% Using parfor for parallel processing
parfor i = 1:N
data{i} = readmatrix(strcat(datafiles(i).folder, '/', datafiles(i).name));
end
% Since waitbar updates are not possible inside parfor, consider alternative progress indication
disp('Data Import Complete');
delete(gcp('nocreate')); % You may choose to delete the parpool
The limiatation to this is that, you will not be able to update the "waitbar" as you are running all it parallely. You might also need to ensure that you have enough RAM to store all the ".csv" files. From your description, the files alone seem to be over 36GBs! The slowdown might also be due to the same reason.
You might want to consider processing the CSVs as a batch.
  2 个评论
Sam Marshalik
Sam Marshalik 2024-6-12
Just wanted to mention that you can use DataQueue to still have a waitbar with parfor or parfeval. You can learn more about it here: Send and listen for data between client and workers - MATLAB (mathworks.com). This will let you read in the files in parallel and still maintain an idea of how many files you have read in vs. how many are left.
George Li
George Li 2024-6-12
Thank you both. I have tried parfor with a pool of 36 processes and it now takes 178s to finish ingesting all the data vs ~20 mins with the original for loop! Thank you for your help this is perfect.
Fortunately this is short enough now that I don’t really need a wait bar anymore but I will be trying out the data queue method you have posted anyway.

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

标签

产品


版本

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by