Artificial intelligence: Danish startup Be My Eyes is developing an app based on the Open AI model that acts as a virtual assistant for people with blindness.
Be My Eyes has a mission: to help blind people explore the world around them using smartphone technology. Created in 2015, the app developed by the Danish startup of the same name connects blind users with a network of sighted volunteers who, through a video call, ‘lend’ their eyes to observe and describe what they see through the device’s camera. In the not-too-distant future, however, volunteers may no longer be needed. Be My Eyes is developing a beta version of the application that relies on a virtual assistant based on GPT-4, the latest version of Open AI’s artificial intelligence model. GPT-4, unlike its predecessor GPT-3.5, is a multimodal system: as input, it can accept not only text but also images, the contents of which it can scan to contextualize them in response to a written command.
The function developed by Be My Eyes with this capability is called, without too much imagination, Virtual Volunteer. Instead of video-calling a live volunteer, users can send a picture to the AI-based virtual one, which then answers users’ questions about the picture’s content. “We have only recently had access [to GPT-4] but have already seen unparalleled performance compared to any existing image and object recognition system,” said Michael Buckley, CEO of Be My Eyes. “The implications for global accessibility are profound. In the not too distant future, the blind and visually impaired community will use these tools not only for visual interpretation needs, but also to achieve a greater degree of independence in their own lives.”
GPT-4
The developers of Be My Eyes began testing the inclusion of GPT-4 in the app in early February. The results and the feedback from beta testers have been so positive that they have convinced the startup to speed up the release of the new function, which could be open to all users in the coming months. The virtual assistant will not replace humans but will be one of the options available in the application.
“What sets the Virtual Volunteer tool apart from other image interpretation technologies available today is context, with a deeper level of understanding and conversational capabilities never before seen in the field of digital assistants,” the startup explained in a post introducing the new feature. “For example, if a user sends a picture of the inside of their fridge, the Virtual Volunteer will not only be able to correctly identify the contents, but also extrapolate and analyze what can be prepared with those ingredients. The tool can also offer a series of recipes and send a step-by-step guide on how to prepare them”.
