The mask has a separate workspace, so what you need to do is get the values from the base workspace before using them.
There is a function called evalin which will let you evaluate an expression from a separate workspace. In this case, we want to grab a and b from the BASE workspace (where you defined a and b) and stick them in the mask's workspace.
An example of code that works for the mask display callback:
myVal = evalin('base','abs([a b])');
fprintf('(%4.2fx-%4.2f)',myVal);
- Sebastian
