Quickly finding maximum value of a large (~8,000,000) vector

1 次查看(过去 30 天)
I was trying to see if I could speed up one of my functions so I ran it through profiler. The script loads big files and writes out smaller portions of the data for me to use later. Before writing however, I check to see if the data has any abnormally large values and if it does, I skip it. After looking at the profiler output, I saw that checking to see if there are any abnormally large values is what takes the most time by far (for 70 files, about 450 seconds out of 550 seconds). I never would have thought would have been the slowest part of the code so I'm wondering if I'm missing something that I could easily do to speed it up.
So here's the abridged version of the code:
for i = 1:nFiles % loop over files
Data = load_data(file(i)); % data is a structure with metadata and the actual
% data is in Data.trace which is 8640001x10x3
for s = 1:10 % loop over stations
badStation = 0;
for c = 1:3 % loop over components
% if one component is bad, skip the station
if max(abs(Data.trace(:,s,c))) > badThreshold
badStation = 1;
break;
end
end
if ~badStation
for c = 1:3 % write loop
write_data(Data.trace(:,s,c);
end
end
end
end
I've checked that memory isn't an issue as there's still 9 GB free during run time. Anything simple I can do to improve this?
Thanks in advance.

采纳的回答

José-Luis
José-Luis 2013-1-22

更多回答(2 个)

Walter Roberson
Walter Roberson 2013-1-22
You can vectorize a bit:
for a = 1 : 10
if max( reshape( abs(Data.trace(:,s,1:3)), [], 1) <= badThreshold
for c = 1 : 3
write_data(data.trace(:,s,c));
end
end
end

Jan
Jan 2013-1-24
编辑:Jan 2013-1-24
See FEX: anyExceed : This can test the values of the array on the fly, such that abs(data) is not created as a temporary variable. The testing stops early, when any element outside the wanted range is found. Therefore a positive test can be found without searching the complete array.

类别

Help CenterFile Exchange 中查找有关 Data Type Identification 的更多信息

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by