It looks like the polygons are 2D surfaces in a 3D environment. Wouldn't that give us a line of intersection rather than an area? If the rectangles are laying flat on the same plane then their overlap would produce an area and that would relatively easier to solve since you could reduce the dimensionality to 2D.
Perhaps a more concrete example would be helpful. It shouldn't be difficult to put together some values that produces the two objects so we can reproduce their coordinates on our end.
An image would also be helpful.
