To confirm:
Inside MATLAB, you are using system() to run R.
Inside R (that is running inside MATLAB), you are using commands that are intended to start a new MATLAB session that is independent of the one that was used to system() out to R ??
But if so, then why are you using lsof to find a port? I get the impression that you are expecting R to be able to connect to the original MATLAB session by using an existing port that MATLAB happens to be listening on (for something) ? But you do not mention anything like having started MATLAB originally with a named pipe and a ptty that might be able to feed commands into it through standard input, so I have no idea what you think MATLAB might be listening on. Unless inside MATLAB you had already used a command to open a socket ???
I wonder if you should be using Matlab$startServer ?
I do not have any experience with R, but I have unix experienc. I can tell you are using some kind of Linux. The MATLAB graphical interface does listen on various sockets to talk between its component parts (for example the help browser is a different process), but it does not listen for external commands. The MATLAB command line (without the graphic interface) does not listen for external commands, but can be fed stdin if you open the command line directly without the graphic interfac.