x=0:.1:10;y=sin(x);h1=figure(1);h2=plot(x,y);
condition=@(x,y) abs(y)<.1;
h3=brush(h1);set(h3,'Color',[0 .5 1],'Enable','on');
set(h2,'BrushData',condition(x,y))
Thanks to Sarah Wait Zaranek. The code above works with little error -- color of data is red instead of greenblue. The goal consists in marking the points of plot with ordinate less that const.
Some remarks on my understanding. 1) Command "brush" creates brush-object(h3) and adds the field "BrushData" into line-object 2) Values of BrushData may be integer but non-zero values aren't distinguished 3) set(h3,'Color'..) change the icon of BrushTool on Figure Editor but not affect on color of marked points