While xlsread is no speed demon, I've never had that kind of slowdown even with large worksheets on that order. Which Matlab release/OS/Excel version out of curiosity; mayhaps somebody will know something that might be release-specific.
Other than that, "rolling your own" to do direct COM interaction with Excel or save the data from Excel in text format (CSV, say) and reading it that way would be the alternatives. I'd personally try to eliminate Excel entirely from the equation if possibly could and generate the data for Matlab in much more amenable form.
