At Google I/O 2023, the American giant unveiled the future of video games: Project GameFace. It is a PC program that allows users to control video games with facial expressions. The idea came about to help an avid gamer, Lance Carr, continue to pursue his hobby despite being diagnosed with muscular dystrophy. Carr plays from a PC and has a whole range of specialized accessories and peripherals to help him control his system, but they were lost due to a house fire. Carr can no longer move around much independently, so video games, especially MMORPGs like World of Warcraft, are a very important channel to stay in touch with the rest of the world.
Therefore, a team of developers at Google intervened with a decidedly innovative solution that does not require any specific hardware beyond a webcam. The software developed by the company makes it possible to transform the user’s face into a grid of dots and ‘translate’ the movement of these dots into mouse actions, such as pointer movement, clicks, and much more.
“After hearing Lance’s story, we knew we had to help. So we collaborated with Lance to co-design Project Gameface, a hands-free gaming mouse using open-source Google MediaPipe solutions. Head movements and facial gestures captured simply from a webcam are recognized by machine learning models to move the cursor accordingly. To adapt to the needs of different users, we incorporated the idea of a gesture size, which allows users to control how prominent their gesture needs to be to trigger a mouse action,” Google said.
New game possibilities
Project Gameface uses Google’s open-source MediaPipe solution, which links together different artificial intelligence models to give you an output. The output is a mesh of 468 points on your face that are then converted into telemetries such as mouse movements or clicks’, says Laurence Moroney, Lead AI Advocate at Google. Moroney says it works natively on devices, and ‘one of the most amazing things about the technology is that it only requires input from a webcam’. Google explains that the software is based on a series of AI models – a theme that practically pervaded every company announcement at the I/O 2023 developer conference.
In addition to having great potential for other people in the same condition as Lance Carr (and not only for playing games but also for writing and performing other tasks on the PC), it is easy to see the future of this technology in other scenarios – even just the animation of one’s avatar in a video game, or why not, in virtual reality. The project has been released in open-source form: anyone can download and consult it.
Suppose Google is not the first of the technologies designed to transform the face or parts of a person’s body into game peripherals; certainly, the project is the only one today that is truly open and available to developers. In the past, Intel had presented its RealSense 3D camera to allow developers to scan 78 different points of a face and recreate facial and emotional expressions. Now nine years away, Intel’s technology never really broke through, partly because it had perhaps arrived too early.
