The primary reason that I can think is the "stride of 4" in the first convolution layer. Having such a huge stride reduces the amount of information for the next layer drastically, which will eventually prevent the network from learning some valuable features.
If the reason behind this design choice is to reduce the output activation size, it would be better acheived with gradually reducing the size over multiple convolution-relu layers.
There can be many other reasons behind the accuracy saturating at around 75% and here are some things that can be tried to avoid it.
- Train the network for more number of epochs.
- Vary the learning rate, starting with a larger number (1e-2) and gradually decreasing it after every n(5-10) epochs. The following weblink talks about how tto do this: Vary training Options for DL Network
- There aren't enough parameters in the network. Adding more neurons in the pen-ultimate fullyConnected layer can help solve this issue.
Hope this helps
