Issue with ncread when _FillValue, add_offset, and scale_factor are present
显示 更早的评论
I have run into an issue when trying to concatenate multiple netcdf files with the same variables into one file, combining these variables.
First, I create a new netcdf file using the header information from one of the files I wish to concatenate using ncinfo and ncwriteschema, changing the length of the variable along which I wish to concatenate to unlimited.
When I read a variable that does not have the attributes _FillValue, add_offset, and scale_factor using ncread, the variable reads into my workspace as I would expect, and writes as expected into the new file.
However, when I read a variable that does have those attributes, the variable reads into my workspace seemingly without applying those attributes, contrary to ncread's documentation. I checked this using netcdf.getVar, and the results are the same between the two methods, which seems to indicate that ncread is failing to apply the attributes.
Strangely, however, when I use ncwrite to store the variable data in the new netcdf file, and then read it back in with either ncread or netcdf.getVar, I get something different! The variable data now oscillates between the _FillValue and -1*_FillValue.
Hypothesis: To me, this seems to indicate that ncwrite is trying to apply the attributes, but since ncread or netcdf.getVar did not apply them inversely, it's now applying them a second time! This causes the values to be larger in magnitude than the _FillValue, and confuses the netCDF file.
An easy workaround would be to apply the attributes myself using ncreadatt and ncwriteatt, but I'm not sure my hypothesis is correct, so I'm hesitant to brute force the process in that way.
Working on attaching code and small enough netCDF files, will update when those are ready.
UPDATE: netCDF files are uploaded as "netcdf_files.zip", matlab code uploaded as "forumpost_ncread.m".
1 个评论
Daniel Holstein
2023-11-29
This is a few years old, but I am having a similar issue. When reading a .nc into matlab using netcdf.getVar the fill values are correct as defined in the file. If loading with ncread, all fill values are changed to NaN. I believe that netcdf.getVar is reading the matrices as they are stored, but ncread converts any value that equals a fill value to NaN... this can be an issue for manipulating these files, as the fill values can change...
采纳的回答
meghannmarie
2020-12-2
编辑：meghannmarie
2020-12-2
Ncread is working, put a breakpoint on line 108. When you use ncread and have a scale factor/add offset, it returns a double. In your code, you are taking your double variable and recasting it an int16 which is turning everything into integers. You need to delete lines 108 and 109.
vname = info.Variables(kn).Name;
var = ncread(fget,vname); %variable that is double precision when there is a scale factor/add offset
dtype = info.Variables(kn).Datatype;
var = cast(var,dtype); %You are now recasting double to int16, delete this
4 个评论
Sam Kastner
2020-12-2
编辑：Sam Kastner
2020-12-2
meghannmarie
2020-12-2
It looks like it made my files correctly. Only thing I see wrong is it puts February before January. To fix that, you need to sort your flist variable.
What problem are you still having?
When I try this the numbers seem to be correct:
files = {'ECMWF_1hr_2018_Jan.nc';'ECMWF_1hr_2018_Feb.nc'};
for f = 1:numel(files)
d = ncread(file,'u10',[1 1 1],[1 1 1])
ncid = netcdf.open(file);
varid = netcdf.inqVarID(ncid,'u10');
i = netcdf.getVar(ncid,varid,[0 0 0],[1 1 1]);
add_offset = netcdf.getAtt(ncid,varid,'add_offset');
scale_factor = netcdf.getAtt(ncid,varid,'scale_factor');
d == double(i) * scale_factor + add_offset
netcdf.close(ncid)
end
Sam Kastner
2020-12-2
meghannmarie
2020-12-3
If you add the lines below to your code, you will see ncread is working just fine. Apply the offset and scale factor and then look at your comparison pictures. Without applying those, you are comparing apples to oranges (int16 numbers to doubles).
ncid = netcdf.open(flist(1).name); % open original netcdf file for reading
varid = netcdf.inqVarID(ncid,'u10'); % get wind speed variable id, should be 3
u_longway = netcdf.getVar(ncid,varid); % get wind speed variable data
add_offset = netcdf.getAtt(ncid,varid,'add_offset');
scale_factor = netcdf.getAtt(ncid,varid,'scale_factor');
u_longway = double(u_longway) * scale_factor + add_offset;
netcdf.close(ncid);
ncid = netcdf.open(fname); % open new netcdf file for reading
varid = netcdf.inqVarID(ncid,'u10'); % get wind speed variable id, should be 3
u_longway_new = netcdf.getVar(ncid,varid); % get wind speed variable data
add_offset = netcdf.getAtt(ncid,varid,'add_offset');
scale_factor = netcdf.getAtt(ncid,varid,'scale_factor');
u_longway_new = double(u_longway_new) * scale_factor + add_offset;
netcdf.close(ncid);
更多回答（0 个）
类别
在 帮助中心 和 File Exchange 中查找有关 NetCDF Files 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
