reading a very large number of csv files

Question

Amir Ahmadi 2019-4-1

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/453778-reading-a-very-large-number-of-csv-files

评论： Amir Ahmadi 2019-4-2

Hi Everyone,

I have 100k csv files they are named file-0001, file-0002,...file-9999, file-10000, ... , file-99999, file-100000. I want to read each one at a time, manipulate and store results in matrix for plotting purposes using a single loop. I'm using the code below to make a structure and read them one by one using a loop. Everything is fine until I reach the transition in number of the digits from 4 to 5 and 5 to 6 (i.e. file-9999 to file-10000 and file-99999 to file-100000) at these points MATLAB reads the files improperly and gives me distorted graphs. Would you please help me with an efficient way to acomplish this task?

Thank you

d=dir('file-*.csv');
n=length(d); 
for i=1:n
  data =csvread(d(i).name, 1, 0);  
  p = data(:,4); % extract the property of interest and overwrite in each iteration
  norms(i) =  norm(p); % manipulate and store
end

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

per isakson 2019-4-1

编辑：per isakson 2019-4-1

Have you checked whether the problem is caused by the data/format of the files in question?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2019-4-1

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/453778-reading-a-very-large-number-of-csv-files#answer_368545

在 MATLAB Online 中打开

N = 100000;
norms = zeros(N,1);
for i = 1 : N
    filename = sprintf('file-%04d.csv', i);
    data = csvread(filename, 1, 0);
    norms(i) = norm( data(:,4) );
end

%04d is a decimal format that always uses at least 4 digits for the integer, using leading 0's if needed. More digits will be output if needed, so it will smoothly go from file-0001 to file-0009 to file-0010, and so on, using leading zeros, and eventually will go from file-9999.csv to file-10000.csv using 5 digits when needed, and eventually file-99999.csv to file-100000.csv

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Amir Ahmadi 2019-4-2

Thank you very much it helped a lot.

请先登录，再进行评论。

Answer 2

dpb 2019-4-1

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/453778-reading-a-very-large-number-of-csv-files#answer_368547

Try

https://www.mathworks.com/matlabcentral/fileexchange/10959-sort_nat-natural-order-sort

and see if it will solve your problem.

MORAL: Always ensure you have chosen a wide-enough field for such naming conventions! You could write a script to rename them such that they will sort correctly in ASCII sequence and I'd certainly recommend if your script creates output files that it be able to handle as large of a number as you can possible imagine getting--or create a different naming scheme.