How to split huge .csv files to multiple .csv files based on size?

Question

Anjan 2018-11-20

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/431078-how-to-split-huge-csv-files-to-multiple-csv-files-based-on-size

编辑： Walter Roberson 2021-2-13

I am trying to split huge .csv files (11 GB) that has both combination of text and numbers into mutiple files based on size (0.5 GB each). I tried using some of the answers in the matlab community but no luck

https://www.mathworks.com/matlabcentral/answers/345534-how-to-import-specific-columns-from-huge-csv-file

https://www.mathworks.com/matlabcentral/answers/322033-how-to-split-excel-file-using-matlab

I hope someone can help!

4 个评论
显示 2更早的评论隐藏 2更早的评论

Walter Roberson 2018-11-21

编辑：Walter Roberson 2021-2-13

Which OS? The easiest way to do this is with the unix split command , quite easy . For Windows I would look at https://www.gdgsoft.com/gsplit/

Anjan 2018-11-25

Thanks Walter, It does work but I had to split many files. The code that user "dbp" recommended in the below comment works great if you want to split files in Matlab.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

dpb 2018-11-21

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/431078-how-to-split-huge-csv-files-to-multiple-csv-files-based-on-size#answer_348269

在 MATLAB Online 中打开

fidR=fopen('originalfile.csv','r');                 % open the big file to read
NFiles=20;                                          % number of files to create
NPerFile=round(Nrows/NFiles)+1;                     % rough number records / file
for i=1:NFiles
  try
    fidW=fopen(num2str(i,'fileNew%00d.csv'),'w');   % open a file to write
    for j=1:NPerFile
      fwrite(fidW,fgets(fidR));                     % transcribe lines verbatim
    end
    fidW=fclose(fidW);                              % close that one
  catch
    fidW=fclose(fidW);                              % close that one
    fidR=fclose(fidR);
end

is a poor-man's split tossed off at the console here; the rounding for number of records per file should catch the whole file; the error routine should only occur when the last file runs into feof on the last one if my logic is right.

And, yes, unfortunately, we can't always make others do sensible things about how they collect data we're subsequently given, granted...I recognized that was likely the case, hence the smiley.

4 个评论
显示 2更早的评论隐藏 2更早的评论

dpb 2018-11-26

编辑：dpb 2018-11-27

在 MATLAB Online 中打开

Ah, yes, "air code" is fraught with typos...

fidW=fopen(num2str(i,'fileNew%02d.csv'),'w');

is correct format string, not '%00d' and you had already defined NRows above... :)

NB: Using

sprintf('fileNew%d.csv',i) % See footnote

will end up with files numbered ...w1,...w2,......w9,...w10,...w11,...w12,...,...w19,...w20,...,

which will come back in alphanumeric order with dir, not in natural order so you'll get ...w1,...w10,...w11,...w12,...,...w19,...w2,...w20,......w9,...,...

probably not the desired result. The '%02d' adds the leading zero so alphanumeric and natural sort order are the same.

(*) To be clear, it's not the difference between sprintf and num2str, it's the format string. I tend to use num2str because it has been vectorized to create an array of strings whereas sprintf runs all the vector output strings together as one long string so that takes additional machinations to separate out when used in vector expressions. So, for self-consistency, I just tend to use the one instead of swapping back and forth based on use context. Just a style preference, basically.

Tanumaya Bhowmik 2021-2-13

How will this script work on a .csv file with header? Would it be better to remove the header from the big file, do the split, and then add headers on the top line of each of those split files? How can I concatanate headers as the first line of an entire directory of .csv files? Thank