How to import and read several large csv in matlab

Question

Ricardo Duarte 2022-6-21

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1745220-how-to-import-and-read-several-large-csv-in-matlab

评论： Sulaymon Eshkabilov 2022-6-22

采纳的回答： Sulaymon Eshkabilov

Pasta1.csv

在 MATLAB Online 中打开

Dear all,

I have a huge amont of large csv files (about 170Mb each).

I need to open them and use the information in two of the columns to plot, let's say collumn D and E (see the attachment).

This is what I did so far, however I found that it is not very efficient.

%Load directory
directorio=uigetdir;
x=dir([directorio,'/','*.csv']); 
K=length(x); 
Atotal=[];
tabletotal=[];
fprintf('Leitura: \n')
tic;
for k=1:K 
    FILENAME =  strcat(x(k).folder,'/', x(k).name);
    
    TotalData=readtable(FILENAME);
    table=TotalData(:,[4 7 18 19 27]); %4-vessel name; %7-vessel type; 18-longitude; 19-latitude
    fprintf('File_%s: %s \n', num2str(k), FILENAME);
    
    tabletotal=[table; tabletotal];
    
    %Coluna 2= longitude
    %Coluna 3= latitude
end
loadAIS=toc;
fprintf('Tempo de carregamento do AIS: %s \n', num2str(loadAIS) );
clear table;
clear TotalData;
A = table2array(tabletotal(:,2:5));

Thank you in advance

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

M. A. Hopcroft 2022-6-21

Which part specifically is not very efficient?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Sulaymon Eshkabilov 2022-6-21

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1745220-how-to-import-and-read-several-large-csv-in-matlab#answer_990435

在 MATLAB Online 中打开

If your data file names are sequential, then you may consider using this type of approach here:

FILE = fullfile('C:\Users\...', '*.csv');  % Directory where the files are residing. 
LIST = dir(FILE);
N = length(LIST);                          % N = number of files 
D = zeros(???, ???, N);                    % Memory allocation to speed up the process of data storing
for ii = 1 : N   
    FFName = fullfile(LIST(ii).folder, LIST(ii).name);
    DATA = readmatrix(FFName);                
    D(:, :, ii) = DATA;                    % NOW D contains all data from 10000 files
end
...  % Select and process the part of D that is necessary
    

An alternative way might be this one:

P=pwd;   % If your files are located in the current directory
P = fullfile('C:\...\...', '*.csv'); % Specify: directory where the files are residing. 
S = dir(fullfile(P, '*.csv'));  % Select the file extension to suit your data files
M = 0;
for k = 1:numel(S) 
    F = fullfile(P, S(k).name);
    D = D + readmatrix(F);                
end
...