I found my answer, so maybe useful for someone else as well. I need to apply conjugate symmetry condition for 2D fourier transform, both in magnitude values and phase components. I applied it as below for a n*m matrix:
"H" is the magnitude:
H(1,1) = 0;
H(1,m/2+1) = 0;
H(n/2+1,m/2+1) = 0;
H(n/2+1,1) = 0;
H(2:end,2:m/2) = rot90(H(2:end,m/2+2:end),2);
H(1,2:m/2) = rot90(H(1,m/2+2:end),2);
H(n/2+2:end,1) = rot90(H(2:n/2,1),2);
H(n/2+2:end,m/2+1) = rot90(H(2:n/2,m/2+1),2);
Similar operation but with negative sign applies to "phi" (phase):
phi(1,1) = 0;
phi(1,m/2+1) = 0;
phi(n/2+1,m/2+1) = 0;
phi(n/2+1,1) = 0;
phi(2:end,2:m/2) = -rot90(phi(2:end,m/2+2:end),2);
phi(1,2:m/2) = -rot90(phi(1,m/2+2:end),2);
phi(n/2+2:end,1) = -rot90(phi(2:n/2,1),2);
phi(n/2+2:end,m/2+1) = -rot90(phi(2:n/2,m/2+1),2);
You can refer to this link for a good example of 2D fourier transform on a matrix and how the results should look like: http://fourier.eng.hmc.edu/e101/lectures/image_processing/node6.html