This problem is similar to ones that I have worked in my career in missile development. You have inspection data from aerodynamic surfaces and you want to determine if they are in alignment with the required tolerance. In your case, you want to compute a reference axis that puts all three blades in the same plane. To work a problem like this, I would attempt to use the data points which are most relevant to the problem. Let's assume that all of the inspection points have the same error. It is further assumed that all three blades have the same nominal shape. In order to specify the plane of the propeller blade, the points further away from the hub are more useful, since they contain the most information regarding the angular alignment - i.e. they have the smallest angle per unit error. So I would focus on using points further from the hub.
It is also informative to compare the aerodynamic shapes of the three blades, so I would approach this problem as follows:
1) locate the center of the propeller system, perhaps using x-z cross sections of the hub to identify the center of the hub circle. Adjust the data center to this point and perform all subsequent rotations relative to this point.
2) define a radius, near the propeller tip, and construct the surface profile for each of the three blades along this circle (as in the first figure, below).
3) Each of these profiles is plotted relative to it's blade axis. (as in the left part of the second figure).
4) The objective is to determine two angle rotations (about x and z) that will cause these three misaligned profiles to align with each other. The simplest way to do this is to find the maximum point in each of the three profiles, and compute x and z rotations that put these three points at the same y value.
The advantage to an approach such as this is that it also gives a comparison of the relative shapes of the three blades. If one of the blades is warped, it will be revealed. By reproducing the analysis at different radius values, it will show if any of the blades are bent relative to the others.
