I would suggest what I think is the easiest way:
If MODELFUN is a matlab function handle, use inside your "matlab method" the command str2func , in order to transform a string into a function handle
function out = mynlinfit(X,Y,MODELFUN,BETA0)
MODELFUN2 = str2func(MODELFUN); % MODELFUN is a string. e.g: '@(x) cos(x)'
% MODELFUN2 is a function handle.
out = nlinfit(X,Y,MODELFUN2,BETA0);
Now, MODELFUN2 is a function handle (which can be used in the original nlintfit), while your method's input MODELFUN is a string, which you don't need to cast in C#, as .NET strings are directly interpreted by matlab.
Hope it helps.
