prompt = {'Enter Figure Type:','Enter Output Type:'};
dlgtitle = 'Input';
fieldsize = [1 45; 1 45];
answer = inputdlg(prompt,dlgtitle,fieldsize);
If the user clicked OK, then answer is a cell array of character vectors, e.g., {'0';'1'} if the user entered 0 and 1. You need to convert those character vectors to numeric before validating their values. That's why the comparison is not working properly - you are comparing a character vector, e.g., '0' to a number, e.g., 0.
Also you need to check that those character vectors are not empty, i.e., that the user entered something for each prompt, and you need to handle the case that the user closed the input dialog figure without clicking OK (i.e., clicked Cancel or just closed the dialog), in which case answer is an empty cell array.
if isempty(answer)
% user cancelled or closed
% you decide what it makes sense to do in this case
% maybe return?
else
% user clicked OK
answer = str2double(answer); % convert each character vector to a scalar numeric
% if the user didn't enter anything or entered something that can't be made
% into a scalar number, e.g., 'baba' or '1 2', then that element of answer
% is NaN after doing str2double, so check for NaN here as well as
% checking the values:
if isnan(answer(1)) || answer(1) < 0 || answer(1) > 3
f2 = msgbox("Invalid input, choose a number between 0 and 3", ...
"Error", "error"); % maybe use errordlg instead of msgbox? doesn't really matter
% maybe return here?
end
end
