PosePIE: Replace Your Keyboard and Mouse With AI-Driven Gesture Control

Speaker 1 [00:08]

I'm Daniel. I have a background in computer science and engineering, and I did my Master of Science degree in 2019. I then worked in research for some years, did research on machine learning for communication networks. And since one year, I'm now a senior software developer at Tegmeyer Incubator. And at Tegmeyer Incubator, we do development of smart home technologies. We are located in the heart of Hamburg, and we want to make the smart home, as you know it from science fiction, a reality. So we have a really strong focus on AI, and we want to make a smart home that detects the user and then reacts proactively in behalf of the user to fulfill their wishes. And for this, it's really important to detect where the user is and what they are doing. and we do this partly with normal mono cameras and then we perform a human pose estimation on it. So human pose estimation in general is a computer vision discipline where you want to detect the key points of the human body and then get the positions and the confidence values for each of these key points from a normal RGB image. There are several models for it, such as OpenPose, MoveNet, and the Pose variants of the YOLO V8 and YOLO 11 models. And most of them, in this case, all except for OpenPose, use a 17 key points model, which is defined by the CocoaPose dataset, which is the dataset many of these models are trained on. and the key points you can see on the right-hand side we have for the hat, we have ear key points, eye key points and the nose, then we have the shoulders, the elbows, the wrists and we have the hips, the knees and the ankles. And this is what we do day to day and last year in November we did a hackathon, a two-day hackathon where we ask us two questions. The first one was how can we use this post data to recognize gestures and how can we control an arbitrary game or program with it. And the result of it is postpy. So pi stands for programmable input emulator which is a tool, an open source tool you can all download, which which allows the user to control any game or program using hand or body movements. It's inspired by Glove Pi, which really has a peak right after the Nintendo Wii was released, and a lot of people used it to control applications on the PC with the Wii Remote. And we do something quite similar to this, but you don't need any special hardware for running Postpile, you can just use any off-the-shelf webcam, and that will detect you, and you control your game or program. We emulate keyboard, mouse, and gamepad inputs from it. And you can use Python scripting to map the detected gestures to these inputs. So you can adapt it to your own application. We also support multiple players simultaneously. And we can achieve real-time performance by running the post-estimation model on the GPU. Here's an overview of the architecture. We start with the image from the webcam. Then we perform the pose estimation on it. Then we do some gesture recognition using some handcrafted algorithms. Then we have the Python scripting, where you can write your own script to map these gestures. And then we do the input emulation in the end. For detecting the pose, we use the Ultralytics YOLO framework, which is a framework for running inference and training on the YOLO vision models. There are several pre-trained models for different tasks in this framework, such as for detection, for image segmentation, classification, and oriented bounding box detection, and of course, also pose estimation. YOLO also includes the BotSort and the ByteTrack trackers, which are used to assign unique IDs to individual detections over time. So you can track them from one frame to the other. And we use this to keep track of the individual players when they are in the video. It also supports various backends. You can export the models too, and you can also use them to run the models. And in PostPy, we rely on TensorRT. And TensorRT is an NVIDIA technology where the model is optimized for the GPU by trying to map it to different hardware functions and then profiling it so you can achieve the best performance on each individual GPU. One challenge we had when implementing the gesture recognition was to make it robust against different factors. So you can, for example, have different image resolutions or aspect ratios if you use different webcams. And we solve this problem by normalizing all our image coordinates to an autosquare around the image. So the middle point of our coordinate system is always in the center of the image, and all angles are always kept, no matter what the original aspect ratio of the image is. We also can handle different frame rates. So there are some cameras working with 30 FPS, others are working with 60. And we handle this by implementing all our calculations and filters on time and not rely on the specific frame rate. Another issue is that the size of the person can also vary. So sometimes you have a really large person in the image. Sometimes they are really small. And we solve this by normalizing all the detection algorithms to the shoulder width, because we assume that the player is always standing in front of the camera. So we use this one. And then we use some really strong low-pass filtering to smooth out any small variations. You can see all the gestures we can detect. At the moment, we have six different gestures. We have an arm raising gesture, a leaning gesture, steering, jumping, swiping, and pointing. And you can extend these gestures by just deriving from base class. And you can use the filters that are included in PostPy in order to implement your own gesture detection algorithms. The arm raising gesture is the most basic one. It's just detected if the key point of the wrist is above the eye. And we have separate gestures for each arm. And this one is used to join as a player. So when you start the program, it will not detect you immediately, but you have to raise your arm. And then it will assign you a player, which is really useful if you are in a crowded environment. You have multiple people in the scene, but only some of them want to play the game. then they can join in and get assigned a player ID. Then we have the leaning gesture, which is just the angle of the spine to the vertical axis. And this one is detected by taking the midpoint between the shoulder and the hip key points. And then we calculate the angle, and we can detect it whenever all of these four key points are visible in the image. We then have a steering gesture which is like turning an imaginary steering wheel and we detect this with the wrist key points and just calculate the angle between the key points. And we detect it based on the shoulder width so if you put your hands close together it detects that you are in the steering mode and you can use it for example to accelerate in the game and if you increase the distance between your hands then it will no longer detect it. We also have a jumping gesture which we detect by the upward movement of the hip key points and then we have a threshold when the speed of this movement is above the threshold that will be detected and the threshold is based on the shoulder width to handle the size of the person, and we also have a configurable sensitivity that you can also configure in your individual script to set when you want to detect jumping. And we then use some low-pass filtering to avoid accidental triggers because these key points are always a bit jittering on the image, so you don't want to get the detection triggered by this, and this is why we use low-pass filtering here. We then have a swiping gesture, which is just swiping in each of the four directions with your hand, and we detect it using the risky point, and it's really similar to how the jumping is detected. So we use the speed of the movement and compare it to a threshold based on shoulder width and sensitivity, and also apply low-pass filtering here. And our most complex gesture is a pointing gesture, which you can use as a mouse replacement. So for this you have a box which is centered at your shoulder and then goes to your other shoulder and then within this box the pointing gesture is detected. You can configure the aspect ratio of the box and whenever you hold your hand still for a certain amount of time, then it's a click. We also use a lot of filtering here, so strong filtering for the shoulder position because You don't want to have the mouse position influenced by just your shoulder being slightly moved while you move your arm. And then we use some weak low-pass filtering on the wrist position itself to still make it responsible, but also suppress a bit of the jitter. For the input emulation, we have keyboard, mouse, and gamepad. Keyboard is just pressing each of the keys on the keyboard. You can either press it or not. For mouse, we have two different modes. We have a relative and an absolute mode. Relative mode is what you have with a normal mouse. So the mouse is somewhere on the screen, and then you can set an offset on how much you want to move it. And the absolute mode is more like a touchscreen, so you can directly set where on the screen you want to set the mouse, and then you can also trigger a click event. For the gamepad, we emulate an Xbox 360 controller, which has several analog axes in addition to the digital buttons, which is really useful if you want to input some analog values, some floating point values into a program or into a game. For example, for the steering in a racing game, you can use these analog axes. And we can also emulate multiple of these gamepads, which then gives us the multiplayer support. The input emulation is implemented separately for each operating system. We are currently supporting both Linux and Windows. On Linux, we do all input emulation using the uinput kernel module, and on Windows, we do the keyboard and mouse input via the Windows 32 API and the gamepad input via a special kernel mode driver. For mapping these gestures to the inputs, we have the Python scripting and we have a declarative API for mapping the gestures to inputs, and you can see here in the lower half of the screen, this is an example script as the user would write it. We have a base class where you're derived from, and then this class is a setup and an update function. In the setup, you can, for example, specify the maximum number of players you want to use, and you can add a keyboard, a mouse, or any number of gamepads. And then in the update function, this is what is executed on every frame. There you do the mapping from the gestures to the inputs. You can see here, for example, we just say keyboard, and then the up arrow should be the first person that is detected and logged in. And from this person, it's the right-hand swiping up gesture that we want to map. So whenever it's detected, it just presses the up arrow. And the scripting also gives you direct access to the key point positions and also the confidence values. So you can also write in your script some small logic to detect some other gestures. And you have the ability to use all the filters of PostPy, such as an EWMA filter, and you can also use any other Python module here. So here's a demo of me using these swiping gestures to navigate the YouTube page of the PyCon. And you can see whenever I do the swiping gesture, the selected video changes. Here you can see the pointing gesture in action, working as a mouse, also navigating the PyCon website. You can see I just hold still for a short moment, and then it clicks. So you can navigate the full web page here. Our last demo is a racing game, so here we have TrackMania, and you can see the steering gesture being used here to control the car. We use gamepad emulation to directly map the analog steering angle to the game as an analog value, and this works quite well. getting used to but then you can achieve quite good results with it we we do the acceleration when you put your hands close together so if you don't want to accelerate just increase the distance and then you feel will break of it so in future work we want to extend the post detection also to 3d post data there are some 3D pose models for it, but we can also use multiple cameras and then do triangulation between the detected key points to build a 3D representation of the skeleton. Or we can use AI-based depth estimation. This you can see on the right-hand side. There we run the depth anything model on the image. You can see the hand is a bit more yellow than my So this is then detected just from the 2D image to be in front of me, and with this we can try to reconstruct the whole 3D pose. And of course, being a smart home company, we want to control smart home applications with it. So you can imagine, for example, you have your TV, you can control it with these swiping gestures, or you might have a simple turning knob gesture to increase the volume, decrease the volume, the volume so you don't have to constantly speak to a voice assistant in order to do these fine control movements. In summary, PostPy performs post-estimation on a webcam image using the Ultralytics YOLO framework. We have some handcrafted gesture detectors, and we do the gesture to an emulated input device mapping for our Python scripting. Modify itself is also completely written in Python, so you can completely modify it. And it's open source. It's available on GitHub under the GPLv3 license. So we hope you might try it out, and we will hear from you what you will do with it. Thank you.

Speaker 2 [17:29]

OK. That was a wonderful talk, Daniel. So we have some questions on Slido. So I do want to address it for you. So the first question, does this also work if the user is sitting? If not, how complicated would it be to extend to do it?

Speaker 1 [17:43]

Yes, it works, so as long as the key points that you want to use are visible, you can use it also when sitting, so all the navigation gestures that also work, just when sitting.

Speaker 2 [17:54]

Second question, does dynamically changing background interference with results? If so, have you tried combining any background removal techniques on top of the estimated poses?

Speaker 1 [18:05]

The background typically is not a problem because it directly detects the humans and Puts the key points on all humans, so it doesn't really matter what's in the background. There's no need to separate it by hand

Speaker 2 [18:17]

Do you store the initial values of key points as baseline to calculate all the gestures later?

Speaker 1 [18:25]

I don't really got it I think we we don't really need initial values we did just detect the where the key points are and then we perform the gesture detections so and whenever they are visible we can detect the gestures otherwise we will just say they are not detected so there is no need for initial values

Speaker 2 [18:48]

How do you decide which gesture to select when multiple gestures are detected at the same time?

Speaker 1 [18:54]

they are then just all detected and then you can do in your own script you can maybe write some logic to say which of them has priority but they will just be detected

Speaker 2 [19:06]

How do you manage swiping for a left-handed or ambidextrous user?

Speaker 1 [19:13]

And all gestures are always implemented for right hand and left hand side and they are separate gestures So you can write in your own script which one you want to use you can also use them for separate things Swiping with the right hand for one thing swiping with the left hand for another one

Speaker 2 [19:29]

you mentioned that your calculations are based on time not on the frame rate so what is the time window and does it differ for various sensors

Speaker 1 [19:38]

We just calculate back Everything to a time unit so we calculate the movement speed then in this Relative coordinates or in our normalized coordinate system per time unit and this way we are robust against different frame rates

Speaker 2 [19:55]

In the future, will the system run all the time or only turned on when needed, e.g. via switch?

Speaker 1 [20:03]

Now the idea is to run it all the time or the base system There might be some triggers when you want to actually activate it So if there's no one in the room Maybe you can just use normal motion detectors to deactivate it then but as soon as you're there it should be available

Speaker 2 [20:18]

we just got one again question so this will be the last one so how reliable are the key points delivered by YOLO have you had any issues with them

Speaker 1 [20:27]

In general, they are quite reliable as long as you're clearly visible. There is some small jitter between the frames, but this is nothing too extreme. You can usually handle quite well with some filtering. And if there are some extreme light conditions, then it might get bad, but we never really have huge problems with it.

Speaker 2 [20:47]

Okay, I think that's it. So let's give him a round of applause. Thank you, Daniel.