I understand that the main issue is repeated querying of large number of points to inpoygon function.
One approach can be filtering of points that are too far away from the polygon.
So, you can make two extra polygons like the below image with one extra polygon but with one polygon inside the original polygon:
So, at a time instance you can query only points which are not too far from the original polygon.