When I try to run your Code directly, without any changes. I do not see any error, rather I get a piecewise plot with 20 points.
x = linspace(-10,10,20);
mask = x < 0;
y(mask) = 3* x(mask).^2 + 5*x(mask) - 7;
y(~mask) = 3*x(~mask) + 4;
figure(5);
plot(x, y,'--md','LineWidth',2,'MarkerFaceColor','g','MarkerEdgeColor','g');
title ('Plot Results of Mask');
xlabel ('-10 to 10 20 points');
ylabel ('Different for < 0 & >= 0');
grid on;
Were to running a different script ?