how to append data from mat file.?

3 次查看(过去 30 天)
pruth
pruth 2016-2-15
回答: dpb 2016-2-16
hello. guys... my data is so heavy(4-5 gb data)...it taking so much time for loading ... i have some mat files...they are actuality cell array...files are like brodata.mat <5136 *2>, brodata1.mat <4328*2>,brodata2.mat <4367*2>,brodata3.mat <5328*2>..... i am trying to make only one file (cell array) ..which contain all the variables from these files i.e <19159 *2 >.
  2 个评论
dpb
dpb 2016-2-15
What is in each cell in the cell arrays? If it's just a single numerical element per (as might come from having used textscan without 'collectoutput','true') then using cell2mat and resaving as simply numeric arrays will save quite a bit of room and make loading faster besides...
If not, need to know more about what is in them to have any real definitive suggestions.
pruth
pruth 2016-2-16
in each cell array there are two column , like bellow
<15094x3 double> 735965.041666667
<15072x3 double> 735965.083333333
<15096x3 double> 735965.166666667
<15078x3 double> 735965.250000000
<15101x3 double> 735965.291666667
<15070x3 double> 735965.375000000
<15097x3 double> 735965.458333333
.
.
.
<15106x3 double> 736145.625000000
<15137x3 double> 736145.708333333
<15111x3 double> 736145.791666667
<15133x3 double> 736145.833333333
<15132x3 double> 736145.916666667
i want one file which contain all the data from all cell array. i want next cell arrays two column should append here below these two columns. hope u understand. i m little slow in English.

请先登录,再进行评论。

回答(1 个)

dpb
dpb 2016-2-16
I expect you'll just make your loading time worse but I'd use the matfile object to try this and see how it goes...
Start by making a copy of the first file in a new file so have a way to recover if having a huge file isn't the answer to your problems...
copyfile('brodata.mat','newbro.mat') % make the initial new file
newObj=matfile('newbro.mat','Writeable','true'); % create matfile object as writeable
d=dir('brodata*.mat'); % get the list of files
for 2=1:length(d) % process; skip the base (dir returns ordered list)
data=load d(i).name; % get the data from the file
newObj.data=[newObj.data;data];
clear data % get rid of extra copy
end
In all likelihood the above will run into memory issues as it does create a copy in memory
"data" above is a placeholder for your cell variable name in your files; use whatever it is.
You can -append to a .mat file, but only additional variables with save; if you use the same variable name it replaces that name, it does not add onto that array internally -- illustration:
>> whos -file new.mat
Name Size Bytes Class Attributes
c 4x2 6912 cell
>> c
c =
[100x2 double] [7.3638e+05]
[100x2 double] [7.3638e+05]
>> save new.mat c -append
>> whos -file new.mat
Name Size Bytes Class Attributes
c 2x2 3456 cell
>>
Net result is the 2x2 from memory replace the already existing 4x2 on file.
Only way I can think of to create larger arrays on file w/o holding in memory would be to create stream files (which don't support cell arrays well). Then you could do simple file system copy but you lose the existing data structure.

类别

Help CenterFile Exchange 中查找有关 Large Files and Big Data 的更多信息

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by