Multi-thread parsing and loading thousands of csv files
7 次查看(过去 30 天)
显示 更早的评论
I have a folder with 2500 csv files, each 15MB each. I currently have a script that reads each csv into a cell array container as follows at the bottom.
Unfortunately this serial process takes a very long time to open each csv one by one.
Ideally I would like to multi-thread or open multiple csv files in parallel and save them into either their own set of cell arrays per 'thread' and later combine and sort them, or into one big cell array as it is currently.
%% IMPORT FILES
directory = '\\headnode\userdata\George\ANSTO\ANSTO Day 2\Data\D14\';
datafiles = dir(append(directory,'*.csv'));
N=length(datafiles);
a = 0;
data = cell(1,N);
f = waitbar(a,'Importing Data...');
for i = 1:N
data{i} = read_csv(strcat(datafiles(i).folder, '\', datafiles(i).name));
waitbar(i/N,f);
end
waitbar(1,f);
close(f);
2 个评论
采纳的回答
Ganesh
2024-6-12
编辑:Ganesh
2024-6-12
You will be able to parallelize the process with a "parfor" instead of using the "for" loop. Using parfor will require a "Parallel Computing Toolbox" license. The implementation would look as follows:
%% IMPORT FILES IN PARALLEL
directory = '\\headnode\userdata\George\ANSTO\ANSTO Day 2\Data\D14\';
datafiles = dir(append(directory,'*.csv'));
N = length(datafiles);
data = cell(1, N);
if isempty(gcp('nocreate'))
parpool; % Adjust the number of workers as needed, e.g., parpool(4)
end
% Using parfor for parallel processing
parfor i = 1:N
data{i} = readmatrix(strcat(datafiles(i).folder, '/', datafiles(i).name));
end
% Since waitbar updates are not possible inside parfor, consider alternative progress indication
disp('Data Import Complete');
delete(gcp('nocreate')); % You may choose to delete the parpool
The limiatation to this is that, you will not be able to update the "waitbar" as you are running all it parallely. You might also need to ensure that you have enough RAM to store all the ".csv" files. From your description, the files alone seem to be over 36GBs! The slowdown might also be due to the same reason.
You might want to consider processing the CSVs as a batch.
2 个评论
Sam Marshalik
2024-6-12
Just wanted to mention that you can use DataQueue to still have a waitbar with parfor or parfeval. You can learn more about it here: Send and listen for data between client and workers - MATLAB (mathworks.com). This will let you read in the files in parallel and still maintain an idea of how many files you have read in vs. how many are left.
更多回答(0 个)
另请参阅
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!