You can only export to xls or xlsx if you are using MS Windows with Excel installed. There is no Mathworks provided method to write xls or xlsx on OS-X, no matter whether you have Excel or not. (This is because the export method requires using ActiveX to communicate with Excel.)
You may wish to try out http://www.mathworks.com/matlabcentral/fileexchange/38591-xlwrite--generate-xls-x--files-without-excel-on-mac-linux-win from the File Exchange