The transformation you are using is a simple tranformation into polar coordinates. When you have y in there also, this becomes what is commonly known as cylindrical coordinates. So we will have a transformation of your coordinate system from (X,Y,Z) into (THETA,Y,R). Totally common in mathematics, and often very useful.
A useful tool in this is to just use cart2pol. It will convert the pair (X,Z) into (theta,R). And since you know that Y is unchanged, that will make things almost trivial, simpler even than using atan2 and then computing R.
A problem is, you seem to think there is a problem of a discontinuity. In reality, yes, we will expect to see a discontinuity, in the sense the angle theta will always lie in the (half open) interval [-pi,pi). Or [-180,+180), if you prefer to think in degrees.
That is, you can imagine the rotation being done smoothly in a circle. At some point, the rotation angle starts to approach 180 degrees. Just at the point where it tips over, past 180 degrees say to 181 degrees, the angle reported will revert to -179 degrees. The rotation is the same. Everything is happy. But because the angles as reported only live in the interval [-180,180), you must see what you are describing as a discontinuity, because at that point, you are visualizing this as an angle in a polar coordinate system. You can still have angles in that coordinate system that are larger than 180, or less than -180. But when you compute that rotation angle from X and Z, it will be contained in that limited interval. And that is what you are thinking of as a discontinuity.
(Another sometimes useful tool here is the unwrap function. You really don't need it in this context though.)
But, really, is this a discontinuity in the real world? NO! The rotation matrix you have does not truly convert things into cylindrical coordinates at all. You can think of it as an implicit transformation. All it does is to rotate the X and Z axes around the Y axis, so leaving the Y axis unchanged. If you think of the result after rotation, there is no discontinuity. If you spin a top around, there is no huge bump as it spins. You do not hear a thump, thump, thump,... as it spins. Just a faint whir as it disturbs the air. The discontinuity lies only in your mind as you try to visualize the cylindrical coordinate system.
Hmm. Maybe I need to expain this as an animation. That may take some time to draw pictures. So first, I'll try to draw a mental picture:
Consider a point that rotates around a circle. So imagine this as the point (1,Y,0). Think of that point rotating around the y axis. At a rotation of 90 degrees, it will become (0,Y,1), etc. Eventiually, it gets to the point at 180 degrees, where the coordinates after rotation are now (-1,Y,0). If we rotate just a bit more, just 1 degree more, things are still moving smoothly. There is no huge bump in the night. As I have written it here, as a function of the rotation angle phi, that original point will have rotated to: (cos(phi),Y,sin(phi)). Would you agree with that?
In fact, for ANY smoothly incrementing angle phi, that rotation will be smooth and continuous. Again, no bumps will ever be seen, none felt. As you smoothly change phi, the transformation is smooth and continuous. The only discontinuity you will ever see is when you visualize the transformation in terms of that polar coordinte system. And this is because the rotation angle will always be reported in a finite range like [-pi,pi). So the discontinuity lies only in the mind, not in reality.
I'll hope for now this is sufficient, because it is Sunday morning here and I am feeling far, far too lazy to draw anything more than mental pictures and thought experiments. :)
