permutation in regression learner app
2 次查看(过去 30 天)
显示 更早的评论
I am using statistical and machine learning toolbox to find a best model to predict my variable of interest. After having caculated my best model how can I compute permuation to asses the p value?
2 个评论
Ive J
2021-10-24
how can I compute permuation to asses the p value... What do you mean exactly by computing permutation?
回答(1 个)
Shubham
2024-5-15
Hi Parisa,
To compute permutation tests for assessing the significance (p-value) of your best model's performance in MATLAB, using the Statistical and Machine Learning Toolbox, you can follow a general approach. This involves shuffling the labels or responses of your dataset multiple times and recalculating the model's performance for each shuffle. By comparing the original model's performance against the distribution of performances from these permutations, you can estimate how likely it is to observe your model's performance by chance.
Here's a step-by-step guide to performing a permutation test:
1. Fit Your Best Model
First, fit your model using the original dataset. This involves selecting your predictors (features) and response variable, then training the model accordingly.
% Assuming X are your predictors and Y is your response variable
bestModel = fitlm(X, Y); % Example for a linear model, adjust according to your model type
% Compute the performance of your model, e.g., R-squared, RMSE, accuracy...
originalPerformance = bestModel.Rsquared.Ordinary; % Adjust this metric as needed
2. Permutation Test
To perform the permutation test, you will shuffle the response variable Y multiple times, refit the model for each shuffled dataset, and compute its performance metric.
numPermutations = 1000; % Number of permutations
performanceShuffled = zeros(numPermutations, 1); % Preallocate array for performance metrics
for i = 1:numPermutations
% Shuffle the response variable
Y_shuffled = Y(randperm(length(Y)));
% Fit the model to the shuffled dataset
modelShuffled = fitlm(X, Y_shuffled); % Adjust for your model type
% Compute the performance metric for the shuffled model
performanceShuffled(i) = modelShuffled.Rsquared.Ordinary; % Adjust metric as needed
end
3. Compute the P-value
After obtaining the distribution of performance metrics from the shuffled datasets, you can compute the p-value as the proportion of times the shuffled models' performances equal or exceed the performance of the original model.
pValue = sum(performanceShuffled >= originalPerformance) / numPermutations;
Notes
- The choice of performance metric (e.g., R-squared, RMSE, accuracy) depends on your model type and the nature of your prediction task (regression or classification).
- Adjust the model fitting function (fitlm in the example) according to the type of model you're using (e.g., fitglm for generalized linear models, fitctree for decision trees, etc.).
- A low p-value (typically <0.05) suggests that the observed model performance is unlikely to be due to chance, indicating a potentially significant relationship captured by the model.
This approach provides a non-parametric way to assess the significance of your model's predictive ability, complementing traditional statistical tests and confidence intervals that assume specific data distributions.
0 个评论
另请参阅
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!