What is n?
%X = cos(20*pi*5*n*1/100) - sin(pi*2*n*1/100);
%Y2 = 0.765*X - 1.53*(X-1) + 0.765*(X-2) + 1.475*(Y2 - 1) - 0.587*(Y2-2);
% ^^ ^^ ???
Here Y2 appears on the left and on the right side of the assignment. This must fail.
Anyway, this is not a highpass filter at all. Filtering would use the values of X(k-1) and Y(k-1) to determine the output at the point k, not X(k)-1 or Y(k)-1.
Use filter instead:
n = linspace(0, 20*pi, 400); % A bold guess
X = cos(20*pi*5*n*1/100) - sin(pi*2*n*1/100);
a = [1, -1.475, 0.587];
b = [0.765, -1.53, 0.765];
Y = filter(b, a, X);
plot(X, 'r');
hold on
plot(Y, 'b') % You see: only the high frequency passes through
