Yes. Some formats are faster than others. However, which is fastest depends on your operating system and on your exact CPU.
For example, the Intel i3 processor contains special hardware to decode H.264, and Windows 10 Media software knows that and will invoke the hardware, whereas the i5 does not have that hardware (my memory could be wrong about that).
Uncompressed movies take the least amount of time to decode. On the other hand they also take up the most disk space, and fetching from disk is typically the slowest part. Moving the video onto SSD could completely change the balance of which format is fastest.
