Why does Support Vector Regression (fitrsvm) optimization result in poor performance ?

Question

Baran Yildiz 2017-1-6

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/319251-why-does-support-vector-regression-fitrsvm-optimization-result-in-poor-performance

评论： antlhem 2021-5-29

Hi

I am working on a numerical prediction problem (load forecasting). I have a predictor matrix which consists of around 20 numerical variables (X_train: including historical lags, calendar information, temperature etc) and an output vector of real/target load values (Load_train). The data consists around 10.000 points

I am following the below documentation on Support Vector Regression, in particular the section 'Optimize SVM Regression'. https://matlabacademy.mathworks.com/R2016b/portal.html?course=mlml#chapter=4&lesson=7&section=3

However after the exhaustive 'hyperparameter optimization' I get poor prediction performance, especially compared to a simpler SVR which uses Gaussian Kernel, Kernel Scale 'auto' and all other parameters set to default without any optimization. I do standardize the input and output matrices before the training. Please find below the lines for two training procedure:

"Simple SVR"

mdl_simple=fitrsvm(X_train,Load_train,'Standardize',true,
    'KernelFunction','gaussian','KernelScale','auto');

"Optimized SVR"

Mdl_optimized = fitrsvm(X,Y,'Standardize','true','OptimizeHyperparameters','auto',...
    'HyperparameterOptimizationOptions',struct('AcquisitionFunctionName',
    'expected-improvement-plus'))

Would anyone have any advice with regards to the optimization procedure or any ideas why optimized SVR might be giving worse results than simpler SVR?

Best regards

Baran

2 个评论
显示无隐藏无

Walter Roberson 2017-1-6

Generally speaking, when more sophisticated fitting procedures end up giving worse results in practice, the problem can be due to overfitting.

antlhem 2021-5-29

Could take a look into my question? https://uk.mathworks.com/matlabcentral/answers/842800-why-matlab-svr-is-not-working-for-exponential-data-and-works-well-with-data-that-fluctuates?s_tid=prof_contriblnk

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Don Mathis 2017-1-8

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/319251-why-does-support-vector-regression-fitrsvm-optimization-result-in-poor-performance#answer_249761

编辑：Walter Roberson 2017-1-8

在 MATLAB Online 中打开

Maybe it's because the optimized SVR uses the default kernel function, which is 'linear'. If you include 'KernelFunction','gaussian' in your second call to fitrsvm, as you did in your first, it might perform better.

Secondarily, you could try running the optimization longer by adding the field 'MaxObjectiveEvaluations' to your struct:

Mdl_optimized = fitrsvm(X,Y,'KernelFunction','gaussian','Standardize',true,'OptimizeHyperparameters','auto',... 'HyperparameterOptimizationOptions',struct('AcquisitionFunctionName', 'expected-improvement-plus', 'MaxObjectiveEvaluations', 60))

2 个评论
显示无隐藏无

Baran Yildiz 2017-1-24

Hi Don Thanks for the answer. I had previously included gaussian as the Kernel Function type. But your suggestion of running the optimization longer with 'MaxObjectiveEvaluations' argument worked this time. The results are slightly better than the non-optimized SVR... For my problem, i am not sure if this slight improvement worths the extra time to spent on the optimization.

Cheers Baran

Don Mathis 2017-3-23

在 MATLAB Online 中打开

I just noticed your reply. If you have some time on your hands you could try

    Mdl_optimized = fitrsvm(X,Y,'OptimizeHyperparameters','all', 'HyperparameterOptimizationOptions',struct('MaxObjectiveEvaluations', Inf, 'SaveIntermediateResults',true))

This optimizes over 6 hyperparameters until you press Control-C, at which point you'll find an object called BayesoptResults in your workspace. The best hyperparameters are obtained using

bestPoint(BayesoptResults)

which you will then need to pass to a new call to fitrsvm manually.

请先登录，再进行评论。