speeding up the data analysis

10 次查看（过去 30 天）
Federico
Federico 2025-7-28
评论： Federico 2025-8-2
I am working on a project where I have to analyse high frequency wind data (50 Hz) coming from a wind turbine data measurement system. this amount of data must be converted from .dat binary files to .mat files which I can use in matlab. The data must then be filtered and then averaged over 10 minutes to be compared to the data from another measurement system. Doing all this requires the analysis of thousands of data and it's very time consuming (right now it is about 105 seconds just for the data of 2 days). How can I speed the process up?
for ii = 3:length(filedir)
filename = filedir(ii).name;
newdata = ReadFamosDataIntoTimeTable(filename);
% filter
IsConsidered = newdata.avbladeangleGRe<40 ... % normal operation
& newdata.RAWS9>0.5 ... % good availability
& ~isnan(newdata.iv10mswindspeed2GRe) & newdata.av100msabswinddirectionGRe>180 & newdata.av100msabswinddirectionGRe<250;
TurbineData(ii-2).date = filename;
TurbineData(ii-2).iv10mswindspeed2GRe = mean(newdata.iv10mswindspeed2GRe(IsConsidered));
TurbineData(ii-2).ivactivepowerGRe = mean(newdata.ivactivepowerGRe(IsConsidered));
TurbineData(ii-2).CalculatedAirdensity_GRe = mean(newdata.CalculatedAirdensity_GRe(IsConsidered));
TurbineData(ii-2).HWShub1 = mean(newdata.HWShub1(IsConsidered));
TurbineData(ii-2).av100msabswinddirectionGRe = mean(newdata.av100msabswinddirectionGRe(IsConsidered));
end
The function ReadFamosDataIntoTimeTable is to convert the .dat binary data into .mat data

请先登录，再回答此问题。

采纳的回答

Steven Lord
Steven Lord 2025-7-28
What does the Profiler indicate is the section of code that takes the most time?
Does the Code Analyzer app or the code analyzer information included in the Editor when you edit these files make any recommendations that suggest improved performance?
I suspect that ReadFamosDataIntoTimeTable is the bottleneck (simply because it's doing file I/O) but you haven't showed us the code of that function so we can't offer any specific suggestions about it.
One potential (likely small) optimization would be to index into the timetable array once using IsConsidered rather than indexing into each variable separately. Compute something like:
newdataFiltered = newdata(IsConsidered, :);
TurbineData(ii-2).iv10mswindspeed2GRe = mean(newdataFiltered.iv10mswindspeed2GRe);
TurbineData(ii-2).ivactivepowerGRe = mean(newdataFiltered.ivactivepowerGRe);
% etc
  2 个评论
Federico
Federico 2025-8-2
Ok I just checked and for some reason it is faster with the filtering of the single variables (timefine = 119.28) than with the filtering of the whole matrix (249.9). I will now share the code for ReadFamosDataIntoTimeTable:
function TT = ReadFamosDataIntoTimeTable(FileName)
% read in data and get time
Channel = importfamos(FileName);
nt = min(cat(1,Channel.length)); % lowest number
Idx = find(cat(1,Channel.length)==nt,1,'first');
dt = str2double(Channel(Idx).dt(1:end-1));
time = [0:nt-1]'*dt;
% create a time table
TT = timetable();
% loop over channels
nChannel = length(Channel);
for iChannel = 1:nChannel
% data
ThisLength = Channel(iChannel).length;
if ThisLength == nt
Data = Channel(iChannel).data;
else % interpolation on lowest resolution
ThisDt = str2double(Channel(iChannel).dt(1:end-1));
ThisTime = [0:ThisLength-1]'*ThisDt;
ThisData = Channel(iChannel).data;
Data = interp1(ThisTime,ThisData,time);
end
% add to time table object
TT = addvars(TT,Data);
% add units and name
TT.Properties.VariableNames{iChannel} = Channel(iChannel).name;
TT.Properties.VariableUnits{iChannel} = Channel(iChannel).yUnit;
end
% add more time information at the end to avoid to add them for each data
TT.Time = seconds(time);
TT.Properties.StartTime = datetime(Channel(Idx).t0);
end
Federico
Federico 2025-8-2
also I checked the profiler and the most time consuming part of the code is the importfamos function so if you want I can send that too

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Tables 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by