You can build a multitask model using a pretrained CNN by removing the last few layers and replacing them with your custom layers serving the task you want to serve.
Here is an example in case you want to replace the last few layers to make a classification and regression model:
net = resnet50;
lgraph = layerGraph(net);
% Remove the last layers specific to the original task
lgraph = removeLayers(lgraph, {'fc1000', 'fc1000_softmax', 'ClassificationLayer_fc1000'});
% Add new task-specific layers
% Task 1: Classification
numClassesTask1 = 10;
classificationLayers = [
fullyConnectedLayer(numClassesTask1, 'Name', 'fc_task1')
softmaxLayer('Name', 'softmax_task1')
classificationLayer('Name', 'classification_output')];
% Task 2: Regression
regressionLayers = [
fullyConnectedLayer(1, 'Name', 'fc_task2')
regressionLayer('Name', 'regression_output')];
lgraph = addLayers(lgraph, classificationLayers);
lgraph = addLayers(lgraph, regressionLayers);
lgraph = connectLayers(lgraph, 'avg_pool', 'fc_task1');
lgraph = connectLayers(lgraph, 'avg_pool', 'fc_task2');
options = trainingOptions('sgdm', ...
'MiniBatchSize', 32, ...
'MaxEpochs', 10, ...
'InitialLearnRate', 0.001, ...
'Shuffle', 'every-epoch', ...
'Plots', 'training-progress', ...
'Verbose', false);
Hope this helps!
