How to split huge .csv files to multiple .csv files based on size?
15 次查看(过去 30 天)
显示 更早的评论
I am trying to split huge .csv files (11 GB) that has both combination of text and numbers into mutiple files based on size (0.5 GB each). I tried using some of the answers in the matlab community but no luck
I hope someone can help!
4 个评论
Walter Roberson
2018-11-21
编辑:Walter Roberson
2021-2-13
Which OS? The easiest way to do this is with the unix split command , quite easy . For Windows I would look at https://www.gdgsoft.com/gsplit/
采纳的回答
dpb
2018-11-21
fidR=fopen('originalfile.csv','r'); % open the big file to read
NFiles=20; % number of files to create
NPerFile=round(Nrows/NFiles)+1; % rough number records / file
for i=1:NFiles
try
fidW=fopen(num2str(i,'fileNew%00d.csv'),'w'); % open a file to write
for j=1:NPerFile
fwrite(fidW,fgets(fidR)); % transcribe lines verbatim
end
fidW=fclose(fidW); % close that one
catch
fidW=fclose(fidW); % close that one
fidR=fclose(fidR);
end
is a poor-man's split tossed off at the console here; the rounding for number of records per file should catch the whole file; the error routine should only occur when the last file runs into feof on the last one if my logic is right.
And, yes, unfortunately, we can't always make others do sensible things about how they collect data we're subsequently given, granted...I recognized that was likely the case, hence the smiley.
4 个评论
Tanumaya Bhowmik
2021-2-13
How will this script work on a .csv file with header? Would it be better to remove the header from the big file, do the split, and then add headers on the top line of each of those split files? How can I concatanate headers as the first line of an entire directory of .csv files? Thank
Walter Roberson
2021-2-13
compose() or the undocumented sprintfc() are good at creating separate outputs.
更多回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Environment and Settings 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!