Hi Mosalam,
Hyperparameter tuning is the process of selecting the best set of hyperparameters for a learning algorithm. There are several methods for hyperparameter tuning such as Grid Search, Randomized Search and Bayesian Optimization.
While hyperparameter tuning is essential, you can also make an educated guess for the best optimizer based on the nature of your problem and the strengths of each optimizer. For example, ADAM is well-suited for problems with sparse gradients and SGDM is often preferred for large-scale and non-convex optimization problems.
Furthermore, to compare the performance between different optimizers you can use each optimizer to train the model separately. Ensure that other hyperparameters (like learning rate, batch size, etc.) are consistent across different optimizers. You can compare different performance metrics such as accuracy, F-score or loss for different optimizers. Please refer to the following page to understand how to monitor deep learning training progress - https://www.mathworks.com/help/deeplearning/ref/trainingoptions.html#:~:text=auto%27%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20Acceleration%3A%20%22auto%22-,Monitor%20Deep%20Learning%20Training%20Progress,-Stop%20Training%20Early
