How can I move my mouse according to a centroid value?

Question

Priyana Kumar 2021-1-14

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/717490-how-can-i-move-my-mouse-according-to-a-centroid-value

回答： Avni Agrawal 2024-5-16

I have been trying to complete a gesture recognition mouse control program in MATLAB and want to do this without using color detection. So far I have been able to structure the code how I best see fit but I am unable to get the mouse to move with the centroid of my hand.

The gestures I have are number-based:

1 - the mouse will move according to the centroid of my hand.

2 - left click

3 - right click

4 - double click

5- scroll

I have gesture 1 as the start of a while loop with gestures 2-5 inside the loop. Is there any way I can move the mouse according to my centroid value?

I have tried using color detection, which works well, but it isnt an option for my final project. So far, the finger detection has worked for gestures 2-5 but not for 1. Any help is greatly appreciated!

close all
%% Import Java Robot
import java.awt.Robot;
import java.awt.event.*
mouse = Robot;
 
%% Create video input object. 
cam = videoinput('macvideo');   %OR ('winvideo');
cam.Name = 'FaceTime HD Camera'; %change your camera name if needed
cam.TriggerRepeat = 10000;  %change to Inf when needed
cam.FrameGrabInterval = 1;
 
picture = getsnapshot(cam);
frameSize = size(picture);
videoplayer = vision.VideoPlayer('Name', 'Hand Gesture Recognition', 'Position', [100 100 [frameSize(2), frameSize(1)]]);
set(cam,'ReturnedColorspace','rgb')
 
%% Count Number of Fingers
runloop = true;
while runloop
    img1 = getsnapshot(cam);
    img1 = rgb2gray(img1);
    img2 = imcomplement(imbinarize(img1));
    img3 = imfill(img2, 'holes');               %filling all holes
    img4 = bwareaopen(img3, 10000);             %removing objects lesser than 10K size
    
    SE = strel('disk', 10);                     %defining structural elements
    SE1 = strel('disk', 50);
    SE2 = strel('disk', 60);
    img4e = imerode(img4, SE1);                 %image erosion
    img4d = imdilate(img4e, SE2);               %image dilation
    imgfo = img4-img4d;                         %getting fingers
    imgfo(imgfo==1)=0;
    imgfo = logical(imgfo);
    imgfo = bwareaopen(imgfo,5000);
    CC = bwconncomp(imgfo);                     %connected component analysis
    nof = CC.NumObjects;                        %getting number of fingers
    imgfog = uint8(255.*imgfo);
    nofs = num2str(nof);
    imgfogrgb = insertText(imgfog, [0,0], nofs, 'FontSize', 30, 'BoxColor', 'white', 'BoxOpacity', 1, 'TextColor', 'black');
    step(videoplayer, imgfogrgb);
    runloop = isOpen(videoplayer);              %checking if videoplayer is on or off
    set(gcf, 'Position',  [100, 100, 500, 400])
    movegui('northeast')
    pause(2);
  
%     preview(cam);
%     movegui('northeast');
    
%% Begin Move Movements
while nof==1 %move
     st = regionprops(imgfo, 'BoundingBox', 'Centroid');
     imshow(imgfo)
     hold on
     for k = 1 : length(st) 
     bbox = st(k).BoundingBox;
     cbox = st(k).Centroid;
     rectangle('Position',bbox,'EdgeColor','r','LineWidth',1 )
     plot(cbox(1),cbox(2), '-m+')
     a = text(cbox(1)+15, cbox(2), strcat('X:  ', num2str(round(cbox(1))), '    Y: ', num2str(round(cbox(2)))));
     set(a, 'FontSize', 12, 'Color', 'white');
     %hold on;
     end
     X = cbox(1);
     Y = cbox(2);
     set(0,'PointerLocation',[X Y]);
     centroids = cat(1,st.Centroid);
     %mouse.mouseMove(1600-(centroids(1,1)*(5/2)),(centroids(1,2)*(5/2)-180)); 

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Walter Roberson 2021-1-14

set figure or axes CurrentPoint rather than root PointerLocation

https://www.mathworks.com/matlabcentral/answers/429274-get-axes-currentpoint-stack-on-not-updated-current-point#answer_347137

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Avni Agrawal 2024-5-16

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/717490-how-can-i-move-my-mouse-according-to-a-centroid-value#answer_1458816

在 MATLAB Online 中打开

Hi !@Priyana Kumar,

I understand that you are tryinh to move the mouse cursor based on the centroid of your hand, ensure you map the centroid coordinates from the image space to your screen resolution and use the Java `Robot` class effectively. Here's a concise guide:

1. Calculate the Screen Position: Scale the hand's centroid position from the image coordinates to your screen's coordinates.

2. Use Java `Robot` to Move Mouse:

% Assuming 'cbox' is the centroid [X, Y] of your hand in the image
screenSize = get(0, 'screensize'); % [left, bottom, width, height]
screenWidth = screenSize(3);
screenHeight = screenSize(4);
imageWidth = size(img1, 2); % 'img1' is your image matrix
imageHeight = size(img1, 1);
% Scale centroid to screen resolution
scaledX = round((cbox(1) / imageWidth) * screenWidth);
scaledY = round((cbox(2) / imageHeight) * screenHeight);
% Move mouse
mouse.mouseMove(scaledX, scaledY);

Ensure you correctly detect the centroid `cbox` of your hand for this to work. Adjust the scaling as necessary to match your screen's resolution and the camera's field of view.

I hope this helps.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

How can I move my mouse according to a centroid value?

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

回答（1 个）

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

How can I move my mouse according to a centroid value?

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

回答（1 个）

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

0 个评论
显示 -2更早的评论隐藏 -2更早的评论