Currently there is no support for H264 encoding for IPCam. The video stream has to be encoded in Motion JPEG (mjpeg) format. As the video is encoded in H.264 format (not currently supported by IPCAM), it is throwing an error. I have brought this issue to the notice of the concerned people and it might be considered for a future release.
A possible workaround for this issue is to read the network stream with a third-party application (e.g. VLC Media Player) and transcode it to an MJPEG stream. Then try to open that MJPEG stream in MATLAB.
There is a FileExchange submission that might be useful as well: HebiRobotics/HebiCam. HebiRobotics/HebiCam is one of the several submissions in MATLAB File Exchange on MATLAB Central which is a forum for our product users to interact, exchange information and knowledge, without MathWorks' involvement. You may contact the author of this submission directly for specific questions about the implementation.