Hi,
I am unsure about what you want to classify in the video using the optical flow estimation workflow. As per my understanding the above code estimate the direction and magnitude of the movement of any pixel in two consecutive frames. So essentially you can use the algorithm to tell if any object which consists of many pixels is moving or not.
For this you may use any of the pair (Vx,Vy) or (Orientation,Magnitude) as a reference to check if an object is moving. These two pairs are related as for every pixel in the image,
Orientation = atan(Vy./Vx);
Magnitude = sqrt(Vx.^2 + Vy.^2);
So basically, you may cluster the points which have same (Vx,Vy) in a region that might represent an object because for a single object its velocity for every point will remain same.
I would suggest saving any of the pair (Vx,Vy) or (Orientation,Magnitude) for dimensionality reduction as that will bring all the data available from optical flow estimation.
