it is expected that it would be slower.
The decoder does not build a frame index starting from the beginning of the file.
mp4 files are not indexed by frame number: they are indexed by frame time. But they are potentially variable-rate, so knowing the current frame time and the reported frame rate does not always give enough information to calculate a frame time directly. Even files that are not variable rate cannot count on the reported frame rate, as the reported frame rate is considered an approximation. For example, 30 frames per second might be reported for NTSC, or 29.9 frames, when NTSC is 29.97-something frames.
The internal code needs to keep a buffer of the current GOP (group of frames) when moving forward; for MPEG-2 the needed buffering was more restricted. But the code discards the buffer and moves forward from the beginning of the file whenever you move backwards. Potentially it could avoid doing that if the requested frame was within the current GOP. Skipping forward from the current frame does not require going back to the beginning -- the code knows the current frame number and so scan skim forward just figuring out how many frames are there without decoding the details until it arrives at the correct GOP.
Potentially you might be able to do better moving in reverse if you ran forward first building an index of frame numbers to frame times, allowing you to set the frame time (which is indexed) instead of asking for a frame number (which is not.)
Anyhow: what you can do to improve performance is position backwards by several groups of frames, and read several frames, and release them in reverse order, so that you reduce the amount of seeking within the file that is needed.
