how to append data from mat file.?
3 次查看(过去 30 天)
显示 更早的评论
hello. guys... my data is so heavy(4-5 gb data)...it taking so much time for loading ... i have some mat files...they are actuality cell array...files are like brodata.mat <5136 *2>, brodata1.mat <4328*2>,brodata2.mat <4367*2>,brodata3.mat <5328*2>..... i am trying to make only one file (cell array) ..which contain all the variables from these files i.e <19159 *2 >.
2 个评论
dpb
2016-2-15
What is in each cell in the cell arrays? If it's just a single numerical element per (as might come from having used textscan without 'collectoutput','true') then using cell2mat and resaving as simply numeric arrays will save quite a bit of room and make loading faster besides...
If not, need to know more about what is in them to have any real definitive suggestions.
回答(1 个)
dpb
2016-2-16
I expect you'll just make your loading time worse but I'd use the matfile object to try this and see how it goes...
Start by making a copy of the first file in a new file so have a way to recover if having a huge file isn't the answer to your problems...
copyfile('brodata.mat','newbro.mat') % make the initial new file
newObj=matfile('newbro.mat','Writeable','true'); % create matfile object as writeable
d=dir('brodata*.mat'); % get the list of files
for 2=1:length(d) % process; skip the base (dir returns ordered list)
data=load d(i).name; % get the data from the file
newObj.data=[newObj.data;data];
clear data % get rid of extra copy
end
In all likelihood the above will run into memory issues as it does create a copy in memory
"data" above is a placeholder for your cell variable name in your files; use whatever it is.
You can -append to a .mat file, but only additional variables with save; if you use the same variable name it replaces that name, it does not add onto that array internally -- illustration:
>> whos -file new.mat
Name Size Bytes Class Attributes
c 4x2 6912 cell
>> c
c =
[100x2 double] [7.3638e+05]
[100x2 double] [7.3638e+05]
>> save new.mat c -append
>> whos -file new.mat
Name Size Bytes Class Attributes
c 2x2 3456 cell
>>
Net result is the 2x2 from memory replace the already existing 4x2 on file.
Only way I can think of to create larger arrays on file w/o holding in memory would be to create stream files (which don't support cell arrays well). Then you could do simple file system copy but you lose the existing data structure.
0 个评论
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Large Files and Big Data 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!