In the callback for your button, just get the radio button states and do different algorithms according to what was selected
rad1 = get(handles.radio1, 'value');
rad2 = get(handles.radio2, 'value');
rad3 = get(handles.radio3, 'value');
if rad1
% Run algorithm 1
elseif rad2
% Run algorithm 2
elseif rad3
% Run algorithm 3
end
