Hello,
I am using Support vector regression to develop a model and for my data set to reduce the cross validation losses am using bayesian optimization technique. here i am using automatic hyperparameter tuning is using also am using 5-fold and 10-fold cross validation loss minimization. i tried with and without optimization technique. am getting goot model after optimizing the parameters but when i used the tuned parameter train the complete data set after am getting good result when am validate with test data set. but when i check the respective cross validation loss for this final trained model it is not matching with the loss obtained during optimization. both the loss difference is too high for 5-fold but when i continue this for different folds 7, 9 and 10 folds am getting low CV losses also these losses not matching with the losses obtained during hyperparameters that minimize five-fold cross-validation loss by using automatic hyperparameter optimization.
Please any one help me regarding this. If required what ever the code and functions am using i'll share. please help regarding this how can i select the best model and what ever the step i followed is right or wrong below i given the steps followed to develop a model.
1) i trained the model without partitioning the data and later i used same data set for validation. during optimization also i used 5-fold and 10-fold cross validation loss minimization for the complete data set.
2) first i partition the data set into training and testing next i used trained set set to develop a model with and without hyperparameter optimization techniques later i used testing set to validate the model. but in this case when i used 5-fold cross / 10-fold cross validation without optimization am getting high error but when i used bayesian optimization for hyperparameter tuning. After the optimization am with the help of parameter i trained the complete model and tested with testing data set but when i used the cross validation for this data set again am getting large errors. Similarly i followed the steps for both 1st and 2nd step from 2 to 10 folds each part am getting high losses after optimizing the model for respective model but when i compare the models with above two section 7, 9 and 10 fold cross validation losses able to reduce the cross validation losses when i used tuned hyperparameters.
