You say there are two time domain plots, but I see three. Each of the three time domain plots shows an input and an output. I You say "The inputs are the 3 input vectors stacked vertically, and the 3 output vectors stacked vertically." I would not do this, since it creates a signal which may have discontinuities at the boundaries, and those discontinuities in the output would probably look different if the entire concantenated input signal had been run thorugh the system.
To get a coherence estimate with small error bars, I recommend using 8 or more segments, not three. I say that because coherence is estImated from ratios of cross-power and auto-power spectrum estImates. The power spectrum esitmates have variances on the order of (signal power)/sqrt(number of segments). With three segments, the power spectrum estimate variance is on the order of (signal power)/1.73, which can lead to large error bars on the coherence, when using real data.
I am attaching some files of simulated data I made for coherence testing years ago. Each file has two columns: x (input) and y (output). Fs=1000 Hz; number of samples = 1024. The input (x) is the same in all the files. The input is band-limited white noise (30 to 300 Hz). The output equals the input plus uniform noise. The noise is uniform on [-a.+a]. The noise amplitude a is different in ther different files, and the actual random values of the added noise are different. The noise amplituides and files are:
File a=noise half-amplitude
coh1_01xy.txt 0.1
coh1_03xy.txt 0.3
coh1_10xy.txt 1.0
coh1_20xy.txt 2.0
We expect the coherence to be lower when there is more noise on the output, and we exect large error bars on coherence estimates at frequencies where there is little power in the input - because when there is little or no power in the input.
I reocmmend using a Hamming window with half-overlapped segments.
data=load('coh1_10xy.txt');
x=data(:,1); y=data(:,2);
[cxy,f]=mscohere(x,y,hamming(nseg),round(nseg/2),nseg,Fs);
grid on; xlabel('Frequency (Hz)')
title('Coherence'); ylabel('\gamma^2')
Code above produces plot below:
If we plot the coherences for all 4 files, we get
Note that when the noise amplitude (a) is larger, the estimated coherence is lower, as expected. The estimated coherence is low and variable at the frequencies where there is little input power (0 to 30 Hz and >=300 Hz), as expected.