Jiali - the error message is telling you that the handles input parameter is not a structure. If I create a simple GUI with an axes and uitable, define the createFcn for both and run the GUI, then when the createFcn is called, the handles structure is empty for both controls. I suspect that the same is true for you too.
Typically, I don't use the createFcn for controls and instead place all of my initialization code in the OpeningFcn of the GUI. This function is called after all the createFcn calls have been made for each control. For your case, you could then read in the data from the Excel file and use it to populate both the table and the axes. Something like
function untitled_OpeningFcn(hObject, eventdata, handles, varargin)
% Choose default command line output for untitled
handles.output = hObject;
% Update handles structure
guidata(hObject, handles);
% read the data from file
[num,txt,raw]=xlsread('C:\Program Files\data_example.xls');
% update the table
set(handles.uitable1,'data',num,'ColumnName',txt);
% update the plot
plot3(handles.axes1, num(:,1), num(:,2), num(:,3));
Note how easy it is to update the two controls. Try it and see what happens!
A couple of things concerning your original code:
handles.uitable1=hObject;
is unnecessary since hObject is identical to handles.uitable1. I also recommend that you never do this (modifying graphics handles) because that will lead to unexpected/undesired behaviour in your code.
And
handles.axes1=plot3(tab_data(:,1),tab_data(:,2),tab_data(:,3));
guidata(hObject,handles.axes1);
Again, don't do this. You are replacing the handle to the axes control with the handle to the plot graphics object. The call to guidata will then destroy the handles structure, replacing it with a graphics object handle. If you want to save the handle to the plot, then do instead
handles.hPlot3 = plot3(tab_data(:,1),tab_data(:,2),tab_data(:,3));
guidata(hObject,handles);
Check the call to guidata - we just pass handles as the second input parameter.
