Hi,
In my understanding for define custom loss function you need to change the last layer of the model which can be a classificationLayer or a regressionLayer. For defining custom loss function, you need to declare a custom layer which required declaring forward loss and backward loss for backpropagation. So, if calculation of derivative of the loss function is feasible you may declare that in backwardLoss otherwise you may only declare the forwardLoss using the functions that supports dlarray.
Calculating the derivative of loss at each layer for backpropagation is not required as only last layer is the one which is using something custom. For defining the custom layers for different problem statements, you may refer to the following. Note that for using above functions you require Deep Learning Toolbox also.
