You want to remove outliers by year separately, but it's not clear whether each file corresponds to one year of data or whether the code would need to read all the files and then parse the data by year and operate on each year separately. Your code reads (up to) one file (the dir() call will only ever return at most one file), but then there is a loop to read multiple files, so I'll assume you actually have multiple files with data and that there is not necessarily a one-to-one correspondence between years and files. In that case, you can accumulate all the data together and then calculate the outliers in each year separately. Something like this:
% collect time and tide data from the xlsx files in the current directory:
filehtm = dir(fullfile(pwd(),'*.xlsx'));
time = [];
tide = [];
for ii = 1:numel(filehtm)
temp = xlsread(filehtm(ii).name);
time = [time; temp(:,1)];
tide = [tide; temp(:,2)/1000];
end
% go through each year spanned by time and set the outlier tide data to NaN:
YYYY = fix(time/1000000);
uY = unique(YYYY);
for ii = 1:numel(uY)
idx = YYYY == uY(ii);
temp = tide(idx);
Q1 = prctile(temp,25);
Q3 = prctile(temp,75);
IQR = Q3 - Q1;
upper = Q3 + 1.5*IQR;
lower = Q1 - 1.5*IQR;
temp(temp > upper | temp < lower) = NaN;
tide(idx) = temp;
end