You cannot rely on file size to tell you whether data is missing. Current MATLAB .mat files may contain compressed data. You should use "whos -file" on the file to look to see what variables are in the file and their sizes.
Regarding "Saving the data"?
2 次查看（过去 30 天）
显示 更早的评论
Dear All,
I am getting data from NETCDF files as follows:
time=double(netcdf.getAtt(ncFileId,netcdf.getConstant('NC_GLOBAL'),'time'));
data1=...
dat2=..
..
etc
And then save it like this:
data=[time data1 data2..];
data=double(data);
And then it will save as follows:
save(matfile,'data')
But, the data that came out from this is not completed. I mean the size of each matfile that save in this way is around 80k. While it should be ~3000-5000k. So, I am missing some data.
Does anyone knows how to resolve my problem?
I would greatly appreciate your help, Ara
采纳的回答
Walter Roberson
2017-1-17
20 个评论
Ara
2017-1-17
I did use"whos",
159984 struct
and I also checked data from workspace. I have all the variable that I want but when I plot all of the 365 days it shows a little data. So, I checked the size with those that I downloaded from profile data and save it exactly like this.
p=netcdf.inqVarID(ncFileId,'time');
time=netcdf.getVar(ncFileId,p);
Where I save it exactly like above (global data). It comes with many data. That is why I am thinking perhaps the way I am saving the data is not suitable for Global data (?).
Ara
2017-1-17
I have save all the data in one matfile. Is that possible the loop that I wrote is not correct?
for k = 1:365
matFile = sprintf('2013.%03d.mat', k);
if exist( matFile, 'file' )
data = load( matFile, 'data' ) ;
fprintf( 'Load data from MAT-File : %s ..\n', matFile ) ;
else
fprintf( ' MAT-File is not exsist : %s ..\n', matFile ) ;
end
end
and then save it as follows:
matFile= 'Year-2011';save (matFile, 'data');
I think I am not reading all the 365 days, right?
Would you please tell me where the problem is?
Walter Roberson
2017-1-17
You are correct, that loop overwrites data each iteration of the loop.
for k = 1:365
matFile = sprintf('2013.%03d.mat', k);
if exist( matFile, 'file' )
data_struct = load( matFile, 'data' );
all_data(:, k) = data_struct.data;
fprintf( 'Load data from MAT-File : %s ..\n', matFile ) ;
else
fprintf( ' MAT-File is not exsist : %s ..\n', matFile ) ;
end
end
Ara
2017-1-17
Thank you Walter. But, I am getting this error Subscripted assignment dimension mismatch. here is the size data for one day
3555x8 double
But, it does not go through the loop to read all 365 days.
Do you have any idea?
Ara
2017-1-17
Is that possible to read all the matfile in one shot and then save it in another matfile for whole year? In that way I have a data contains all 365 days so that I can easily analyse it without reading through all files each time.
Walter Roberson
2017-1-17
编辑：Walter Roberson
2017-1-17
all_data(:, :, k) = data_struct.data;
The code already loops over all of the files and stores all 365 days of information into the same array, all_data . I did not know anything about the size of the data in the files and had to guess at the shape.
Ara
2017-1-17
Sorry for providing incomplete information, and thanks for your guess. But, it gets same error. I do not think this goes through the loop because when I read only one file the output is data= 3555x8 double
Name Size Bytes Class Attributes
Data 1x1 227696 struct
whos all_data
Name Size Bytes Class Attributes
all_data 3555x8 227520 double
and here is for all 365 days
after reading through the loop 1:365..so this is only the last day 365 whos data
Name Size Bytes Class Attributes
data 1x1 159984 struct
from each matfile I have a "gpsdata", which is structure. Please see bellows the arrangment..Year, day, time, etc
*2013 365 0 9.4 0.0 68.3 139.8 762.0*
Can we concatenate vertically each file? It can read first file and keep it and then concatenate the rest in that one?
Please let me know if I can provide more information.
Walter Roberson
2017-1-17
I did not pre-allocate the output because I did not know if the input data size was fixed. The code I provided creates 3555 x 8 x number of files read, and if you only read one file that would be 3555 x 8 x 1 which MATLAB would automatically show as 3555 x 8.
for k = 365 : -1 : 1 %this has the effect of pre-allocating
matFile = sprintf('2013.%03d.mat', k);
if exist( matFile, 'file' )
data_struct = load( matFile, 'data' );
all_data(:, :, k) = data_struct.data;
all_gpsdata(:, :, k) = data_struct.gpsdata;
fprintf( 'Loaded data from MAT-File : %s ..\n', matFile ) ;
else
fprintf( ' MAT-File is not exist : %s ..\n', matFile ) ;
end
end
Ara
2017-1-18
I understand, no worries. Also, thanks, Walter. But, it does not work again. Here is the error:
Subscripted assignment dimension mismatch.
Cannot display summaries of variables with more than 524288 elements.
It shows nothing that allow me to use the data.
Is that possible to read each file and go to another one and at the end append all the data in one matFile?
Walter Roberson
2017-1-18
for k = 365 : -1 : 1 %this has the effect of pre-allocating
matFile = sprintf('2013.%03d.mat', k);
if exist( matFile, 'file' )
data_struct = load( matFile, 'data' );
all_data{k} = data_struct.data;
all_gpsdata{k} = data_struct.gpsdata;
fprintf( 'Loaded data from MAT-File : %s ..\n', matFile ) ;
else
fprintf( ' MAT-File is not exist : %s ..\n', matFile ) ;
end
end
At the end of this you would have cell arrays all_data and all_gpsdata that contain all of the data read from the files. If all of the entries read are exactly the same length, then you can construct an array of output from them:
nd = ndims(all_data{1});
try
all_data_matrix = cat(nd+1, all_data{:});
catch ME
fprintf('Your data entries are not all the same size!\n');
all_data_matrix = [];
end
nd = ndims(all_gpsdata{1});
try
all_gpsdata_matrix = cat(nd+1, all_gpsdata{:});
catch ME
fprintf('Your gpsdata entries are not all the same size!\n');
all_gpsdata_matrix = [];
end
Ara
2017-1-19
THANK YOU, Walter. You are the best and creative. It works perfectly.
Can I ask another question? I have all variable in all_gpsdata. I transpose and came out like this:
3555x8 double
5403x8 double
5084x8 double
5188x8 double
4563x8 double
4214x8 double
4772x8 double
4590x8 double
Now, I want to plot all the data in column 4 vs column 3, for example. To get these variables for all 365 days. They are not exactly in the same length all my matfiles. So, the array of output did not construct.
Sorry to bother you again.
Walter Roberson
2017-1-19
hold on
cellfun(@(data) plot(data(:,3), data(:,4)), all_gpsdata);
That would be for creating 365 lines. If you wanted to create one single line that had all of the information, then it would look more like
data = vertcat(all_gpsdata{:});
plot(data(:,3), data(:,4))
This assumes that all of the entries have the same number of columns.
Ara
2017-1-19
Wow, thanks. Just using the "cellfun" I got this. Please see the figure; Can we have something like "scatter plot" to avoid this lines?
Index exceeds matrix dimensions.
Walter Roberson
2017-1-19
hold on
pointsize = 10;
arrayfun(@(IDX) scatter(all_gpsdata{IDX}(:,3), all_gpsdata{IDX}(:,4), pointsize, IDX*ones(size(all_gpsdata{IDX},1),1)), 1:length(all_gpsdata), 'Uniform', 0);
hold off
This will color-code each line according to the day number.
If you ended up with some matrices that do not have enough columns, then you need something like
mask = cellfun(@(C) size(C,2), all_gpsdata) < 4;
agps = all_gpsdata;
agps(mask) = {NaN(1,4)};
hold on
pointsize = 10;
arrayfun(@(IDX) scatter(agps{IDX}(:,3), agps{IDX}(:,4), pointsize, IDX*ones(size(agps{IDX},1),1)), 1:length(agps), 'Uniform', 0);
hold off
Ara
2017-1-19
Wow, it looks nice! Though my PC and my MATLAB all hanged out;forced me to restart it several times. So, it could not plot all the days; it shows again "Index matrix dimensions".(?)
Walter, I believe you can solve this problem of mine as well. So, please let me share it with you. I have two type of these matFiles, where I loaded those files as well and the size came out as "1.5G", where it save in another matfile. Now, Its very difficult to load it and plot things as MATLAB shows "out of memory". But, my problem is the "time" in these files come as a second. So, column vector of "time" started from 1 to 9000. Now, I need to match data so that read (let's say column 7) as a function of time (0-24hrs). Do you have any suggestion?
You did help me alot; I really appreciate it.
Walter Roberson
2017-1-19
If you are using a new enough version of MATLAB (might require R2016b) you could look at timetable()
Ara
2017-1-19
Yes, exactly I am using R2016b. Can you please explain more. What should I need to know about the timetable?
Ara
2017-1-20
编辑：Ara
2017-1-20
Thank you for useful documents/links. It helps me to understand. Can I ask you one question? I understand that I can get timetable so that I can arrange time based on hour, which is great. For this, I need to arrange other data based on that as well.
This means that:
Time=seconds(1:3600)'; % this will give me data in one hour. Am I right?
T=timetable(Time, [data in column 1], [data in column 2],...,[data in column 8], 'Variable Names', {'A', 'B',...'H'});
But, my problem is how get 3600 data in column A, as an example, as well? I would know the first 3600 from time represents an hour and then I need to get one hour data, too. So that I can hourly do the analysis.
Can you please give me a hint?
Thank you for all of your great help,
更多回答（0 个）
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Text Files 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!发生错误
由于页面发生更改，无法完成操作。请重新加载页面以查看其更新后的状态。
Translated by 
您也可以从以下列表中选择网站：
如何获得最佳网站性能
选择中国网站（中文或英文）以获得最佳网站性能。其他 MathWorks 国家/地区网站并未针对您所在位置的访问进行优化。
美洲
- América Latina (Español)
- Canada (English)
- United States (English)
欧洲
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom(English)
亚太
- Australia (English)
- India (English)
- New Zealand (English)
- 中国
- 日本Japanese (日本語)
- 한국Korean (한국어)