This is possible to do in MATLAB by using Java libraries.
% Take screen capture
robot = java.awt.Robot();
pos = [0 0 400 400]; % [left top width height]
rect = java.awt.Rectangle(pos(1),pos(2),pos(3),pos(4));
cap = robot.createScreenCapture(rect);
% Convert to an RGB image
rgb = typecast(cap.getRGB(0,0,cap.getWidth,cap.getHeight,[],0,cap.getWidth),'uint8');
imgData = zeros(cap.getHeight,cap.getWidth,3,'uint8');
imgData(:,:,1) = reshape(rgb(3:4:end),cap.getWidth,[])';
imgData(:,:,2) = reshape(rgb(2:4:end),cap.getWidth,[])';
imgData(:,:,3) = reshape(rgb(1:4:end),cap.getWidth,[])';
% Show or save to file
imshow(imgData)
imwrite(imgData,'out.png')
Note: If you are looking to take a screenshot of a figure window, there are many other tools that are much better suited for this, for example the "print" and "getframe" functions, as well as several featured submissions to File Exchange such as "ScreenCapture" or "export_fig".
