Use lower case letters:
[B,A] = butter(4, 0.6);
‘B’ and ‘A’ are the coefficients of the numerator and denominator coefficients of the filter transfer function, respectively.
The cutoff frequency is normalised by the Nyquist frequency, so a normalised frequency of 0.6 means you have a Nyquist frequency of 10 Hz and a sampling frequency of 20 Hz. If those are not your Nyquist and sampling frequencies, you need to redesign your filter.
I do these steps whenever I design a filter:
Fs = ...; % Sampling Frequency
Fn = Fs/2; % Nyquist Frequency
Fc = 6/Fn; % Passband Frequency (Normalised)
Fz = 7/Fn; % Stopgand Frequency (Normalised)
[n,Wn] = buttord(Fc, Fs, 1, 10); % Order Of Filter
[B,A] = butter(n,Wn); % Designs Lowpass Filter By Default
I always use the freqz function at this stage, to be sure the filter is stable. If it is not, I either redesign it or add these additional steps:
[sos,g] = tf2sos(B,A); % Second-Order-Section Implementation
Then use the filtfilt function to filter your signals, to avoid phase distortion in the filter.
