Realistic interactions with distant interlocutors recreated in 3D

Project Starline: Breaking down the spatial barrier to communicate at a distance clearly and immediately is the reason why video meeting applications have sprung up, thanks to which, through a laptop and webcam, we can interact with friends and colleagues no matter where they are. The spike in virtual encounters recorded during the Coronavirus pandemic made Zoom, Meet, Teams and other similar programmes soar in popularity, a useful remedy to the impossibility of seeing each other face to face, despite some limitations that often make it problematic to exchange views in a fluid and effective manner.

How does Starline work

Presenting Project Starline in 2021, Google promised to overcome these shortcomings with a revolutionary technology that would allow us to talk to a distant person in a natural way. A realistic telepresence model obtained by the representation of users in full-scale 3D, which at first glance hinted at the clear breakthrough but also cast doubt on the time required to set up a complicated and expensive system. Two years after its first appearance, Starline today is still a prototype, but it opens up unprecedented scenarios because it has made enormous progress quickly.

The demonstration came during the recent Google I/O, the annual developers’ conference where Big G told the journalists present in Mountain View about the improvements made over the last 24 months, but above all, allowed them to test for the first time the most advanced remote communication technology known today. Seated in front of 65-inch 8K screens, the conversation with a Google employee (who was in another room of the building, but this changes little because the location does not affect the effectiveness of the model) allowed them to be confronted with an interlocutor reconstructed in 3D perceived as very similar to reality.

A Magic Window

Those that Big G defines as ‘magic windows‘, also because they do not involve the use of visors, glasses or other tools other than the screen, allow one to speak, gesture and make eye contact with the other, facilitating communication that flows naturally so much so that several journalists have pointed out the wow effect of the feeling of wanting to touch or shake the other person’s hand. Missing the grip (obviously!) as the technology works, but the actual distance between the speakers is not cancelled.

An exceptional result was obtained by Google with the use of four 120fps cameras placed above and to the sides of the interlocutor, photographed continuously at the beginning to let the artificial intelligence process the images, which then recreates the physical appearance and movements in 3D, showing on the display a body with volume and depth.

Progress and problems to be solved

Exaltation is an automatic reaction to an innovation destined to retire the current standard for virtual meetings. During the two years of tests and the modifications made thanks to the feedback from the employees of the partner companies that experimented with Starline (such as Salesforce, T-Mobile and WeWork), Big G succeeded in achieving a system that, according to the testers themselves, improves the flow and dynamics of the conversation, increases attention to the other person and reduces video meeting fatigue. However, patience will be needed because Google specifies that there is much work to be done, and it will still take years, not to mention that such hardware is likely to be expensive.

The biggest obstacle, however, is figuring out how to manage the computing capacity needed to process the amount of data produced by the cameras. At the moment, everything is done via the cloud and corporate databases. However, Andrew Nartker, General Manager of Project Starline, imagines that later on, “TVs could be easily converted to accommodate the new remote communication technology, but also next computers could handle the data themselves”. Different solutions to solve a crucial knot because Starline should not only be a business tool but an innovation capable of speeding up and improving long-distance interaction in different sectors: doctor-patient, shopkeeper-customer, and grandparent-grandchild pairings are just a few examples of an ideally much longer list.