That happens because the markers of the scatter plot are circle in shape with a finite/comparable size. You can see that by ploting the line y=x, as I have done below -
clc, clear, clf % preliminary settings
[x, y] = meshgrid(-5:0.01:5, -5:0.01:5); % Create a grid of x and y values
expression = x.^2 + y.^2; % Compute the expression x^2 + y^2
mask = expression > 1 & y > x & x <= 2; % Define a logical mask for the region
scatter(x(mask), y(mask)); % Plot the points satisfying the mask
hold on
fplot(@(x) x)
As you can see that the area extends beyond the line, which is not according to the inequality. To tackle this, input a smaller size to the markers of the scatter plot
%Edited code
figure
[x, y] = meshgrid(-5:0.01:5, -5:0.01:5); % Create a grid of x and y values
expression = x.^2 + y.^2; % Compute the expression x^2 + y^2
mask = expression > 1 & y > x & x <= 2; % Define a logical mask for the region
%%Size of the marker specified (the default size is 36)
%%Adjust the value as per requirement
scatter(x(mask), y(mask), 5, 'filled'); % Plot the points satisfying the mask
hold on
% Plotting the boundary lines
% Part x^2 + y^2 = 1, y = x (outside the region) with dashed line
t1 = linspace(-5, -sqrt(2)/2, 100); % x values for the dashed line segment
plot(t1, t1, '--', 'LineWidth', 1.5, 'Color', 'b'); % Plot the dashed line segment
hold on
t = linspace(pi/4, 5*pi/4, 100); % Parameter for the circle segment
x_circle = cos(t); % x values for the circle segment
y_circle = sin(t); % y values for the circle segment
plot(x_circle, y_circle, '-.', 'LineWidth', 1.5, 'Color', 'b'); % Plot the circle segment
t2 = linspace(sqrt(2)/2, 2, 100); % x values for the dashed line segment
plot(t2, t2, '--', 'LineWidth', 1.5, 'Color', 'b'); % Plot the dashed line segment
% Part x = 2 (inside the region) with thick line
y_line = linspace(2, max(y(mask(:))), 100); % y values for the thick line segment
x_line = ones(size(y_line)) * 2; % x values for the thick line segment
plot(x_line, y_line, 'LineWidth', 2, 'Color', 'b'); % Plot the thick line segment
xlabel('x'); ylabel('y'); % Label the x and y axes
title('Domaine de définition'); % Set the plot title
legend('x^2 + y^2 > 1, y > x, x <= 2', 'y = x (outside)', ...
'x^2 + y^2 = 1 (ibid)', 'y = x (ibid)', ...
'x = 2 (inside)', 'Location','best'); % Set the legend
xlim([-5 5]); ylim([-5 5]); % Set the x and y axis limits
axis square % Set the aspect ratio to be square
ax = gca; % Get the current axes handle
ax.XAxisLocation = 'origin'; % Set the x-axis location to the origin
ax.YAxisLocation = 'origin'; % Set the y-axis location to the origin
set(gca,'Layer','top') % Set the grid to be on top of the plot
grid on, grid minor % Display both major and minor grids
hold off % Release the hold on the plot
