Unpredictable execution time of custom function using ncread

Question

N/A 2015-8-3

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/232149-unpredictable-execution-time-of-custom-function-using-ncread

编辑： N/A 2015-8-3

Hello everybody,

I am currently working on my Bachelors Thesis. I am reading medium to large .nc files. These files contain a global data set of wave parameters on a 180 (longitude) x 170 (latitude) grid with a 6 hour temporal resolution.

I have written a couple funtions to make the code more readable for outsiders (the company I'm working for). So I use BA_test.m which contains several sections. Each of these sections contains the needed variables to run the desired funtion; Example:

%%Hs vs Tp plot of any coordiante tuple
tic
if(exist('cnt','var')==0);
    cnt = 1;
end
path = 'E:\Tobias\Daten\';
output_path = 'Z:\Studienarbeiten\Kleine_Tobias\';
kind = 'WAVE'; % WAVE , SEA , SWELL
lon = -37; % deg East; real, phys. coordinate
lat = 46; % deg North; real, phys. coordiante
[cnt] = BA_HS_TP(lon,lat,kind,path,output_path,cnt);
toc

This test program defines the counter variable cnt (used for multiple figures), the input data path, output data path, the kind of variable to be read with ncread (here 'WAVE'), lon, lat.

BA_HS_TP is this:

function [cnt] = BA_HS_TP(lon,lat,kind,path,out_path,cnt)
cur_dir = pwd;
if strcmp(cur_dir,path)==0;
cd(path);
% display(['Cd to ' path])
end
lon = lon + 181;
lat = lat + 81;
switch kind
    case 'WAVE'
        tmp1 = squeeze(squeeze(ncread([path 'HS_1981-2011.nc'],'Hs',[lon,lat,1],[1,1,inf])));
        tmp2 = squeeze(squeeze(ncread([path 'PP_1981-2011.nc'],'WAVE_PP',[lon,lat,1],[1,1,inf])));
        x_string = 'Combined wave peak period [s]';
        y_string = 'Combined wave significant wave height [m]';
    case 'SEA'
        tmp1 = squeeze(squeeze(ncread([path 'SEA_HS_1981-2011.nc'],'SEA_Hs',[lon,lat,1],[1,1,inf])));
        tmp2 = squeeze(squeeze(ncread([path 'SEA_PP_1981-2011.nc'],'SEA_PP',[lon,lat,1],[1,1,inf])));
        x_string = 'Wind sea peak period [s]';
        y_string = 'Wind sea significant wave height [m]';
    case 'SWELL'
        tmp1 = squeeze(squeeze(ncread([path 'SWELL_HS_1981-2011.nc'],'SWELL_Hs',[lon,lat,1],[1,1,inf])));
        tmp2 = squeeze(squeeze(ncread([path 'SWELL_PP_1981-2011.nc'],'SWELL_PP',[lon,lat,1],[1,1,inf])));
        x_string = 'Swell peak period [s]';
        y_string = 'Swell significant wave height [m]';
end 
cmap = load('Histogram_cmap.mat'); % load custom colormap, in order to show the low probability variations in the Hist
tmpp = double([tmp1,tmp2]);
figure(cnt);
hist3(tmpp);
view(70,40);
[n,c] = hist3(tmpp); cnt = cnt + 1;
Anzahl = sum(sum(n));
figure(cnt);
contourf(c{2}, c{1}, n/Anzahl,500);
set(gca,'ydir','reverse');
set(gca,'xaxislocation','top');
shading flat;
caxis([0 1]);
colormap(cmap.map);
cb = colorbar;
ylabel(cb,'relative frequency');
xlabel(x_string);
ylabel(y_string);
% output to excel
xlswrite([out_path num2str(lon-181) 'East_' num2str(lat-81) 'North.xlsx'],n/Anzahl);
cnt = cnt + 1;
end

The problem is, if I run this script from within the BA_test.m (I actually evalute one of several sections), the program runs very slow. Using the "Run and Time" I get around 250-360 seconds of runtime. Running the exact same code from a new script (creating a new .m-file, copy paste) yields a faster execution (~3 seconds). I could understand this by believing there is something dumb I do within the section of the BA_test.m, BUT! When I run this new .m-file for a different lon, lat configuration this new .m-file shows the same, ultra slow execution.

I hope I was able to describe my problem accurately. Of course I have checked the time-consuming code line using the profiler. It turns out if the function runs slow, it is due to the "+netcdf\private\netcdflib (MEX-file)". I googled for anything concerning this topic and only found issues where people were dealing with differences in execution time between for example Matlab and R.

Since this slowdown greatly influences my workflow I would be very very thankful for any hints or potentially useful forums threads!

Cheers Tobias

PS: I have atteched the two files and for your information, the .nc-files to be read are about 11GB, but thats not an issue if I only read specified parts of the .nc-file by using ncread('filename','var_name',[start_x,start_y,start_z],[count_x,count_y,count_z]);

EDIT: added profiler output as pdf.