Hi,
As you said you’re using Fast R-CNN for the first time, the example given on https://www.mathworks.com/help/vision/ref/trainfastrcnnobjectdetector.html#bvkk00v-6 page can help you understand the role of various inputs going into trainingOptions function. There can be many probable reasons behind your network not getting converged.
Since you are training your network using ‘adam’ solver, consider using the other training options associated with it like GradientDecayFactor and SquaredGradientDecayFactor. Also, consider training your network using other methods available in MATLAB R2016a and above, like sgdm and rmsprop. You can learn more about the solver options available to you in the link below:
Also, check to see if the images in your training data do not contain excessive NAN values, as too many NaNs may lead to propagation of NaNs through the network and hence the training will fail to converge.
As far as your initial learning rate in considered, 10^-6 seems to be very small, and with every 10 epochs it is further decreasing. An initial learning rate of 10^-3 may be of help in your case.
