It is possible to use the ActiveX ability, (actxserver('Excel.Application') to start excel) but this option can take a bit of learning and requires a lot more manual control for sheet names and the like.
Basically, what it does is opens Excel within matlab and conducts the operations with the consistently open file. It will still take some time, simply because you have a lot of data, but it will prevent any opening and reopening you may have with tableread or other commands.