For activity recognition, it would not be a good idea to use SIFT or HoG. It would be better to use "signalTimeFeatureExtractor" in MATLAB combined with machine learning to perform the same.
Please refer to the below documentation link on "Human Activity Recognition Using Signal Feature Extraction and Machine Learning". This will assist you with the task at hand.
You can also refer to the below link, which does activity recognition using the Optical Flow data.
Hope you find this information helpful!