Assuming your points are called p1, p2 and p3: One simple approach would be to define a coordinate system (non-orthonormal) with axes (p2-p1) and (p3-p1) as in-plane axes and the cross-product of these axes as third axis. You can then move the point inside this coordinate system and transfrom it in the world system for plotting and further calculations. The z-axis of this triangle-system is perpendicular to the triangle plane.
clearvars
close all
clc
p1 = [8 3 7]';
p2 = [5 9 2]';
p3 = [3 8 1]';
points = [p1 p2 p3];
% Transformation from world frame to triangle frame
% non-orthonormal transformation (normalization possible)
T = [p2-p1 p3-p1 cross(p2-p1,p3-p1) p1; 0 0 0 1];
% Move the point in the Triangle frame and transform to world frame
point_in_triangle_system = [1 1 0 1]'; % the last "1" is for homogenization
point_in_world_system = T*point_in_triangle_system;
% plot triangle and point
fill3(points(1,:),points(2,:),points(3,:),'b','FaceAlpha',0.5)
hold on
plot3(point_in_world_system(1),point_in_world_system(2),point_in_world_system(3),'kx')
If you provide your code, some more specific help is possible; Furthermore, I am pretty sure that Robotics System Toolbox contains functions for the transformation.
