Matlab slowing down while reading netCDF
9 次查看(过去 30 天)
显示 更早的评论
I'm experiencing a problem when I try to read data from a succession of netCDF files. After a certain point (~10 loops of variable y in the example below), the reading operation suddenly slows to a crawl (a few seconds per read -> hundreds or thousands of seconds per read) and never recovers (though it "happily" runs overnight if I let it). The way I resolve it is to save my read-in data frequently, and just kill then restart Matlab to pick up where it got itself into a tizz.
I am currently using customised libraries to read netCDF-4, but I've experienced the same problem in the past with netCDF-3. In the deeper past, and with a much older version of Matlab, I had a faintly similar problem that stemmed from me not closing the files I was reading from, but that doesn't seem to apply here (I'm not even sure if there's a way to close open files now).
About the only thing I've been able to establish is that it appears to be a function of the number of read operations (19 per file below) rather than the number of files being read (336 in total below).
Anyway, is there some basic rule of file management in Matlab that I'm ignorantly flouting? It wouldn't be the first time.
Andrew.
-----
fld2d = zeros([292 362 28 19]);
for y = 1:1:28
for m = 1:1:12
iname = sprintf('EXP500/%d/ORCA1_%dm%dD.nc', 1965+y, 1965+y, m);
for f = 1:1:19
t1 = nc_varget(iname, fld(f,1:flen(f)));
fld2d(:,:,y,f) = fld2d(:,:,y,f) + (t1 / 12);
end
end
end
4 个评论
Ashish Uthama
2011-8-16
It you are in a position to upgrade, version R2010b onwards support netcdf 4 out of the box, no additional set up required.
回答(5 个)
Andrew Yool
2012-2-13
1 个评论
shunping chen
2023-2-23
I also tried clear mex, not getting better. It looks like clear makes no defferent for me.
I tried profile, but I'm not sure if I can get any improvements from the reports.
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/1304880/image.png)
Kelly Kearney
2011-8-16
What version of Matlab are you running? I don't have a clear answer for you, but I experienced some rather dramatic slowdown of the snctools functions when I upgraded to a version of Matlab with the native netcdf support. Though that was more of a across-the-board slowdown, rather than the gradual slowing you describe.
4 个评论
Jerry Gregoire
2012-6-12
If you are not using the 64 bit matlab, on a 64 bit machine your 200GByte Ram will be worthless. ML will only see about 2GB for workspace use.
Ashish Uthama
2011-8-17
I really have no clue, wild guesses:
- Instead of having to restart MATLAB, try issuing a clear mex after a read, how does this impact the next read?
- Does the order of the files being read make a difference? maybe the second file is 'different' in some way?
- Could you elaborate on what you mean by 'customized libraries'? might help some of us try to replicate this.
Jan
2011-8-17
Please use the PROFILEr to find out, which lines needs to most time. Perhaps there is a missing pre-allocation in the function nc_varget, you did not show?
3 个评论
Jan
2011-8-17
See "help profile"... E.g. you can type this in the command line:
profile on
<start your code>
profile report
profile off
Kevin
2013-6-14
I have similar problems. This worked for me. At the end of your loop (whatever it does that accesses files and plots figures...) add the following line just before the return to the top of the loop statement (end):
close all force
This helps!
0 个评论
另请参阅
产品
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!