As you note in the tag, that notation is used most often for the inner product, which is a linear functional, a linear operator from vectors (elements of a vector space) to elements in the scalar field.
Since you also have tagged it with Image Processing Toolbox, I'm assuming you mean the MATLAB function dot().
With N-D arrays, dot() returns the inner products along the first nonsingleton dimension, so that
A = [1 2; 3 4];
B = [1 4; 5 6];
dot(A,B)
Gives a 1x2 vector which are the inner products of the columns of A and B.
While
dot(A,B,2)
gives a 2x1 vectors which contains the inner products of the rows of A and B.
