Francis,
Currently "readXYZ" is a method only on the PointCloud2 message class. The way that it converts the data from the PointCloud2 "data" array into human-usable (x, y, z) coordinates is specific to the PointCloud2 message format and self-description. It would not work even for sensor_msgs/PointCloud, much less an arbitrary encoding that theoretically contains point cloud data.
If your custom message contains point cloud data in the same format as PointCloud2, then my first question would be, "why not just use the built-in PointCloud2 message?" If you are adding additional information to the message, you could easily create a custom message with a field that contains a PointCloud2 message. If you did that, the PointCloud2 message could be used directly with readXYZ or other similar functions in MATLAB.
If you must use the custom message, but it does contain its data in the same format as PointCloud2, you could simply create a PointCloud2 message object and copy the data into it before calling readXYZ.
If your custom message uses a different way of encoding the point cloud data, though, you will need to write your own method for extracting the data into a usable format.
-Cam
