You might want to try writing a batch file wrapper around the program you are calling.
For example, say that you can successfully use redirection in the DOS prompt to respond to your program as follows
program.exe > type response.txt
where program.exe is your program and response.txt contains what you are responding with (the filename, in your case).
You can create a batch file with the following in it (hello.bat): program.exe > type %1
and then within MATLAB, call system('hello.bat response.txt')
You could also use a similar technique with a perl script as well, if you are more comfortable using it, or if you might want more portable code.
Of course, the preferred method would be to change the code that makes the executable to accept command line input rather than requiring an interactive response, which is trickier to script and maintain.
The above example can be tried with notepad and text input file, e.g:
hello.txt: hello there MATLAB user
hello.bat: notepad > type %1
from MATLAB: system('hello.bat hello.txt')
Notepad will then open and have the contents of hello.txt in it.
YMMV with this technique, and it's highly possible that a more sophisticated process would give better results.
