@bahar vojdani - I suspect that you are launching the MFO_toolbox.fig (figure) and not running the MFO_toolbox.m file (in fact, I see the same error Dot indexing is not supported for variables of this type if I double-click or open the figure file and pres the Start Optimization button). There are two files for GUIs created with GUIDE: the figure and the m-file (code). To launch the GUI, you must either type (from the command line)
>> MFO_toolbox
or open the MFO_toolbox.m file in the MATAB file editor and press the greeen run button.
You may have done this already (your log output seems to suggest it) but it isn't clear to me why you are observing the second error Attempt to execute SCRIPT MFO as a function unless you have modified the code. I just downloaded this toolbox from the MATLAB File Exchange and it worked fine for me.