% Your sample data
A = {[100,103,104],[4,5,11],[66],[4,5,1],[85,88,89,77]};
B = {[40,41,41],[4,5,11],[68],[85,88,89,77],[31,66],[1,9,8,7,5],[100,103,104]};
You could do the following
idx = cellfun(@(a) cellfun(@(b) isequal(a,b),B), A,'uni',0);
idx = vertcat(idx{:});
same_cell = A(any(idx,2))
New_A = A(~any(idx,2))
New_B = B(~any(idx,1))
The rows in idx correspond to A and the columns in idx correspond to B. More specifically, rows in idx which contain a 1 corresponds to a cell in A which matched with B, so you can index A(~any(idx,2)) to obtain the non matching cells. Similarly, columns in idx which contain a 1 correspond to a cell in B which matched with A, hence B(~any(idx,1)) gives the non matching cells in B.
