'fitrensemble' doesn't support MAE, but you can use 'TreeBagger' to do what you want. TreeBagger fits a random forest and supports quantile prediction, which can be used to compute MAE. Using MAE doesn't affect how the tree is built, but you can still optimize hyperparameters with MAE as the objective.
TreeBagger doesn't have built-in hyperparameter optimization so you need to use the 'bayesopt' function. Here's a script that does it. Paste it into an editor to run it. Note that it contains a local function definition.
% load a dataset
load carsmall
X = table(Acceleration,Cylinders,Displacement,Horsepower,Mfg,Model_Year,Weight,MPG);
% Define some variables to optimize
Hyperparameters = [optimizableVariable('MinLeafSize',[1,20], 'Type','integer');
optimizableVariable('NumPredictorstoSample', [1,width(X)-1], 'Type','integer')];
% Define the objective function to be the out-of-bag MAE.
fun = @(hparams)oobMAE(hparams,X); % It must be a function of only the hyperparameters.
% Call bayesopt to optimize MAE
results = bayesopt(fun, Hyperparameters);
% Look at the best hyperparameters found
bestHyperparameters = bestPoint(results)
% Fit a "final" model to the full dataset using the best hyperparameters
FinalModel = TreeBagger(300,X,'MPG','Method','regression',...
'MinLeafSize', bestHyperparameters.MinLeafSize,...
'NumPredictorstoSample', bestHyperparameters.NumPredictorstoSample)
% Define a function to fit a random forest of 300 trees and return out-of-bag MAE
function MAE = oobMAE(params,X)
randomForest = TreeBagger(300,X,'MPG','Method','regression',...
'OOBPrediction','on',...
'MinLeafSize', params.MinLeafSize,...
'NumPredictorstoSample', params.NumPredictorstoSample);
MAE = oobQuantileError(randomForest); % Uses the median by default.
end
