Attach a representative dataset and example point to better illustrate.
OTOMH, however, I don't see an obvious solution in general if data groups are not isolated sufficiently from each other that one from group B is closer than the nearest in group A without additional help to restrict which group is desired--the distance alone won't do it.
But, running one of the example codes and looking inside the returned scatter object, the actual jittered coordinates are not exposed nor even visible using Yair's undocumented handles tool, so there isn't an acessible but hidden property to use to retrieve the actual plotted position coordinates of the jittered datapoint; nor could I find an array of jitter values to apply to the fixed x/y values. They have to be inside somewhere, but they do not seem to be accessible at all.
About the best idea I've got is to either look at the specific y-value returned and try to find in which group it resides which is problematical if all datapoints are not unique.
The 'MarkerFaceColor' is 'flat' since uses the colormap so the alternative is to try to match the CData rgb triplet
x=[1:3].*ones(500,3); y=[2 4 6].*randn(500,3)+[0 2 4];
c=eye(3);
hSW=swarmchart(x,y,[],c,'filled')
vertcat(hSW.MarkerFaceColor) % uninformative
vertcat(hSW.CData) % does return the rgb color (r, g, b here...)
