Vectorize a loop to save time
显示 更早的评论
I have a big data set and my current code takes 2 hours. I am hoping to save time by vectorization if that is possible in my case.
I have a table Table with variables ID, t1, tend, p. My code is sth like:
x=zeros(size(Table.ID,1));
for i=1:size(Table.ID,1)
x(i)=sum(Table.t1<Table.t1(i) & Table.tend>Table.tend(i) & abs(Table.p-Table.p(i))>1);
end
So for each observation, I want to find number of observations that start before, ends after and have a p value in the neighborhood of 1. It takes 2 hours to run this loop. Any suggestion?
Thanks in advance!
2 个评论
Walter Roberson
2019-2-4
How are the t1 and tend values arranged? Are tend(i+1) = t1(i) such that together they partition into consecutive ranges that are completely filled between the first and last? Do they act to partition into non-overlapping ranges but with gaps? Are there overlapping regions? Are the boundaries already sorted?
Filip
2019-2-4
采纳的回答
2 hours sounds long. Is the memory exhausted and the virtual memory slows down the execution? How large is the input?
Is this a typo:
x = zeros(size(Table.ID,1))
It creates a square matrix, but you access it as vector obly.
Does the table access need a remarkable amount of time?
n = size(Table.ID,1);
t1 = Table.t1;
tend = Table.tend;
p = Table.p;
x = zeros(n, 1);
for i = 1:n
x(i) = sum(t1 < t1(i) & tend > tend(i) & abs(p - p(i)) > 1);
end
If you sort one of the vectors, you could save some time:
[t1s, index] = sort(t1);
tends = tend(index);
ps = p(index);
for i = 2:n
m = t1s < t1s(i);
x(i) = sum(tends(m) > tends(i) & ...
abs(ps(m) - ps(i)) > 1);
end
Afterwards x has to be sorted inversly. If you provide some inputs, I could check the code before posting. I'm tired, perhaps I've overseen an obvious indexing error.
Is the shown code really the bottleneck of the original code?
更多回答（1 个）
Walter Roberson
2019-2-4
My mind is headed towards creating a pairwise mask matrix,
M = squareform(pdist(Table.p) > 1); %important that Table.p is a column vector
That would be comparatively fast. If the table is very big then it could fill up memory, though.
abs() is not needed for this; pdist will already have calculated distance as a non-negative number.
Now
Mi = M(i,:);
x(i)=sum(Table.t1(Mi)<Table.t1(i) & Table.tend(Mi)>Table.tend(i));
However you should do timing tests against
Mi = M(i,:);
x(i)=sum(Mi & Table.t1<Table.t1(i) & Table.tend>Table.tend(i));
and
Mi = M(i,:);
Tt = Table(Mi);
x(i)=sum(Tt.t1<Table.t1(i) & Tt.tend>Table.tend(i));
2 个评论
Filip
2019-2-4
Walter Roberson
2019-2-4
abs(T.t1 - T.t1.')
would work as a distance function for you in R2016b and later.
类别
在 帮助中心 和 File Exchange 中查找有关 Matrix Indexing 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
