Think of it this way: Each pixel (u,v) in the image are capturing light along some line in 3-D. You can write that line as:
Here 
is the line-of-sight unit-vector of the ray hitting pixel (u,v). The intrinsic parameters determine that function, in the camera coordinate system. What you need to do is to transform (rotate) from the camera-coordinate system to the world system. You should be able to figure out which conversion that should be. (I wrote my own tools for these operations and don't know how matlabs versions work.) Start with checking some simple cases where you can check the algebra by hand, some cardinal directions and rotations of +/- 45, 90 135 degrees in different directions - I find that to be useful to figure out what order things should go.
is the line-of-sight unit-vector of the ray hitting pixel (u,v). The intrinsic parameters determine that function, in the camera coordinate system. What you need to do is to transform (rotate) from the camera-coordinate system to the world system. You should be able to figure out which conversion that should be. (I wrote my own tools for these operations and don't know how matlabs versions work.) Start with checking some simple cases where you can check the algebra by hand, some cardinal directions and rotations of +/- 45, 90 135 degrees in different directions - I find that to be useful to figure out what order things should go.HTH
