You can sum all elements in each column that matches, and divide by the number of nonzero elements:
v = (1:4)'; % your numbers
B = cell2mat(arrayfun(@(x) sum(A==x,1)./sum(A~=0,1), v, 'UniformOutput', false)); % actual calc performed on each element of array v
B = [v, B]; % appending the first column of 1..4 numbers