kfoldfun

Create a cross-validated quantile regression model. Compute the cross-validation quantile loss. Then, compute the quantile loss using training set quantiles instead of predictions.

Simulate 1000 observations from the model $y=1+0.05x+\sin \left(x\right)/x+ϵ$ where:

rng("default"); % For reproducibility
n = 1000;
x = linspace(-10,10,n)';
y = 1 + 0.05*x + sin(x)./x + 0.2*randn(n,1);

Create a 5-fold cross-validated quantile neural network regression model. Use the 0.05, 0.5, and 0.95 quantiles.

CVMdl = fitrqnet(x,y,Quantiles=[0.05 0.5 0.95],KFold=5)

CVMdl = 
  RegressionPartitionedQuantileModel
    CrossValidatedModel: 'QuantileNeuralNetwork'
         PredictorNames: {'x1'}
           ResponseName: 'Y'
        NumObservations: 1000
                  KFold: 5
              Partition: [1×1 cvpartition]
      ResponseTransform: 'none'
              Quantiles: [0.0500 0.5000 0.9500]


  Properties, Methods

CVMdl is a RegressionPartitionedQuantileModel object that contains five trained CompactRegressionQuantileNeuralNetwork model objects (CVMdl.Trained).

Compute the cross-validation quantile loss.

L = kfoldLoss(CVMdl)

L = 1×3

    0.0230    0.0875    0.0229

Each value in L corresponds to one quantile. For example, the first value L(1) is the quantile loss for the 0.05 quantile, averaged across the five folds.

Find the quantile loss when you use training set quantiles instead of test set predictions to compute residuals.

First, create the customQuantileLoss function. The function takes in a compact quantile regression model, training data, and test data, and returns the custom quantile loss. The residuals are defined as the difference between the test set responses and the training set quantiles, instead of the difference between the test set responses and the predicted test set responses.

function loss = customQuantileLoss(CMP,Xtrain,Ytrain,Wtrain, ...
    Xtest,Ytest,Wtest)

    residuals = Ytest - quantile(Ytrain,CMP.Quantiles);
    loss = residuals.*(CMP.Quantiles - (residuals<0));
    loss = sum(Wtest.*loss)/sum(Wtest);

end

To replicate the quantile loss used to compute L, you can use the following residual definition instead.

residuals = Ytest - predict(CMP,Xtest,Quantiles=CMP.Quantiles);

After creating the customQuantileLoss function, pass the function to kfoldfun, along with the cross-validated model CVMdl. Average the results over the five folds.

customL = mean(kfoldfun(CVMdl,@customQuantileLoss))

customL = 1×3

    0.0436    0.2131    0.0484

The customL loss values are greater than the L loss values.