I understand that you want to figure out how 'optimValues.firstorderopt' is computed. I executed a simple example of fmincon along with the a custom ouputput function. Then I placed a break point and stepped in through the different functions that were being called just before my custom output function. It appears that the 'firstorderopt' variable is set in the file 'nlconst.m' at line 952 or 954 inside the 'callOutputAndPlotFcns' to a variable called 'optimerror'. This variable 'optimerror' is set at line 407 and is based on 'normgradLag' and 'normcomp' variables computed at line 404 and 405 respectively of the same file 'nlconst.m'. Using breakpoints and stepping into the file would help find the formula of calculating the 'firstorderopt'.
You can edit the file 'nlconst.m' using the following command
>> edit nlconst.m
Hope this helps.
