You'll get a numeric output from the menu call. Just do a switch-case structure, and calculate the answer using the corresponding function.
choice = menu('Choose an operation','arcsin', 'arccos', 'natural logarithm')
prompt = 'insert an argument: '; % don't need parentheses
arg = inputdlg(prompt); % the argument isn't the answer
arg = str2num(arg{:}); % convert it to numeric from cell
switch choice
case 1
answer = asin(arg);
case 2
% other thing
case 3
% other thing
end
then just display the answer
