The error message is clear, you're using an index greater than the size of the 2nd dimension of an array. The only indexed array in your expression is X1 and your index goes up to 4, so clearly X1 has less than four columns.
Unfortunately, your X1 is global. We strongly advise against using global variables, they're the source of many problems such as this one. It's unclear where X1 is created, nowhere in the code you show so possibly it's never created and hence X1 is empty which would explain the error.
Your fix change the function into a never-ending recursive function, which tries to solve an ode with itself as the ode, so yes it's not going to end well. I'm a bit unclear why you tried that. Your original p5_3fun doesn't involve ODEs at all.
I've not really tried to understand what the rest of your code is doing, just looking at it, you could just replace the original p5_3fun by:
function [ymax,Arms]=p5_3fun(Vht,B,K1,K2, T1, X1) %just add T1 and X1 as inputs
M=345;
dotvy1=(K2/M).*(X1(:,2)-X1(:,1))+(B/M).*(X1(:,4)-X1(:,3));
ymax=max(abs(X1(:,1)));
Arms=sqrt((1/20)*trapz(T1,dotvy1.^2));
end
and add T1 and X1 to the input list every time you call p5_3fun, eg.:
for p=1:length(K2o)
[T1,X1]=ode23s(@(t,x)p5fun(t,x,Vh,B,K1,K2o(p)),[0 20],[0,0,0,0],opts);
[ymax2(p),Arms2(p)]=p5_3fun(Vh,B,K1,K2o(p), T1, X1); %now with T1 and X1 as inputs
end
