append table to a exist csv file

Question

Marco Yu 2016-7-21

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/296611-append-table-to-a-exist-csv-file

编辑： Brent F 2021-8-3

Hi, I have a massive table which need to export to a csv file. The reason of append the table to the file rather grouping the table together and write it as a csv is because: the table size is 242*20X300*300 which is a large table and matlab can't handle the data in the memory.(This could be my computer fault) Anyway, my table is look like this

>> temp_T(1,1:10)

ans =

          patient_ID           R1C1    R1C2    R1C3    R1C4    R1C5    R1C6    R1C7    R1C8    R1C9
    _______________________    ____    ____    ____    ____    ____    ____    ____    ____    ____
    'CMC 009 OD 2008-04-08'    '0'     '0'     '0'     '0'     '0'     '0'     '0'     '0'     '0'

so is it possible to write it to the csv file and then append the data again so for example:

%%this is the csv file

          patient_ID           R1C1    R1C2    R1C3    R1C4    R1C5    R1C6    R1C7    R1C8    R1C9
    _______________________    ____    ____    ____    ____    ____    ____    ____    ____    ____
    'CMC 009 OD 2008-04-08'    '0'     '0'     '0'     '0'     '0'     '0'     '0'     '0'     '0' 
    'CMC 009 OS 2008-04-08'    '0'     '0'     '0'     '0'     '0'     '0'     '0'     '0'     '0'

5 个评论
显示 3更早的评论隐藏 3更早的评论

Christoph Pieper 2020-11-4

Because I was also looking for this: In R2020b there is now an option to append with the writetable function:

'WriteMode','append'

not sure when it was added but 2018b does not have it yet.

Brent F 2021-8-3

编辑：Brent F 2021-8-3

WriteMode='append' is a great idea, but it causes writetable to suppress the column headers.

The fix is to add: `WriteVariableNames=true`

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Peter Perkins 2016-8-3

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/296611-append-table-to-a-exist-csv-file#answer_230685

Marco, it's quite possible (you don't provide enough info to know for sure) that the reason why your table is so big is because you are storing your numeric values in cell arrays of strings. This:

>> x = randn(242*20,300*300);
>> t = array2table(x);
>> whos
  Name         Size                    Bytes  Class     Attributes
    t         4840x90000            3506018448  table               
    x         4840x90000            3484800000  double

is not all that big. Patient_ID is not numeric, but even there you might benefit form using a categorical if the IDs are not all unique.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Guillaume 2016-7-21

3
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/296611-append-table-to-a-exist-csv-file#answer_229343

You cannot use dlmwrite or csvwrite to write tables. You would have to convert the table to a matrix (using table2array). Even then, dlmwrite and csvwrite can only write matrices of numbers and your table contain text, so they're completely out of the equation.

The normal way to write a table to a csv file is to use writetable. Unfortunately, there's no append option.

So, in the end, you can either:

write the tables to different ranges of an excel spreadsheet (with writetable). However, if the merged table uses too much memory in matlab, it's likely that it'll be the same for excel.
write the tables to individual files (with writetable), and merge all these file together afterward.
use low level functions ( fopen, sprintf, etc.) to write your tables.

2 个评论
显示无隐藏无

Marco Yu 2016-7-22

Hi Guillaume. I am interested in the write tables to individual files, but how can I merge all these files together? should I do it in excel?

Or in other hand, is the low level functions is easier?

I only need to transfer the data like above to 1 csv file. Can I have some suggestion please?

Guillaume 2016-7-22

The low levels functions are not particularly complicated, the hardest part would be to figure out the format string for fprintf.

Merging text files together can easily in any OS using the built-in command line tools. On windows, you can use copy. Thus, the code would be something like this:

finalfile = 'hugecsvfile.csv';
tempfileprefix = 'tempfile';  %whatever you want. could include fullpath. file number and extension added in the loop
tempnames = cell(1, numtables); %for stocking file names of temporary files
for tableiter = 1 : numtables
   %...
   %construct temporary table any way you want, e.g.
   temp_t = cell2table(temp_table(2,:), 'variableNames', cellstr(temp_table(1,:)));
   %save table to temporary text file:
   tempnames{tableiter} = fprintf('%s%02d.csv', tempfileprefix, tableiter);  %or any other format you wish
   writetable(temp_t, tempnames{tableiter}, 'WriteVariableNames', tableiter == 1); %only write header for first file
end
%build string for Windows copy command:
copystr = sprintf('copy %s%s %s /b', tempnames{1}, sprintf(' +%s', tempnames{2:end}), finalfile);
system(copystr);
%delete temporary files
delete(tempnames{:});