To create a custom preview window for webcam you can use the following example as a starting point. This code creates a figure, axes, and image elements programmaticaly, but the same configuration would apply for an app created with GUIDE.
%%Example WEBCAM custom preview window for MATLAB R2017a
%%List connected webcams
webcamlist
%%Connect to webcam
c = webcam(1);
%%Setup preview window
fig = figure('NumberTitle', 'off', 'MenuBar', 'none');
fig.Name = 'My Camera';
ax = axes(fig);
frame = snapshot(c);
im = image(ax, zeros(size(frame), 'uint8'));
axis(ax, 'image');
%%Start preview
preview(c, im)
setappdata(fig, 'cam', c);
fig.CloseRequestFcn = @closePreviewWindow_Callback;
%%Local functions
function closePreviewWindow_Callback(obj, ~)
c = getappdata(obj, 'cam');
closePreview(c)
delete(obj)
end
