My quick bet is to vectorize [Y1], [Y2], [X1], [X2].
One question I am not sure about is [Xk]^2 and [X1][X2].
Are they regular matrix multiplication or element-by-element multiplication?
But either way, vectorization still works if I understand your question correctly. For example, your [Y1] is a 2 by 2 matrix. You can vectorize
[Y1]=a1+a2[X1]+a3[X2]+a4[X1]^2+a5[X2]^2+a6[X1][X2]
into 4 equations.
Just write out and compare each element on left and right side of the equations.
