Why would you use symbolic tools to solve this problem? Sorry, but that is a bit silly, using a Mack truck to carry a pea to Boston. A symbolic solution, IF you managed to make it work, would still be slow. That is the nature of symbolic tools.
You have a ray emanating form the origin. You apparently wish to find the intersection of that ray with the edges of a square box.
Algorithm 1:
Use linprog (or any linear programming tool) to find the farthest point along the line, subject to the constraints that the point falls inside all of the edges of the square. This is easy enough to write, and efficient to execute. Of course, it requires a linear programming tool.
Algorithm 2:
Just find the intersection of the ray with each line segment of the box, as a line, so this is just finding the intersection of two lines in two dimensions. If that intersection point falls inside the boundaries of the line segment for the corresponding edge, you are done. Repeat for each edge of the box. This is just a simple loop.
Sometimes a loop is the right solution. Or use linprog.
