The secret to getting augmented reality right? Making smartphones run like a supercomputer

We’ve been waiting for immersive experiences to make it to the mainstream for more than a decade. But, over and over, we’re told to hurry up and wait. Wait for virtual reality headsets to advance to the point where they are as easy to use as the rest of our devices (or at least won’t give us vertigo). Wait for the next innovation in hardware, software, design, or development.

For all of our waiting, immersive experiences will be in the mainstream sooner than most people think, and it’ll be by using devices that have been around for more than a decade: our smartphones. What’s been revolutionized isn’t so much our handheld devices, though. It’s the networks on which they run and the cloud computing that enables a host of applications and services that were never before possible.

“The metaverse is already here and it’s in your users’ pockets,” Will Ochandarena, senior product manager for extended reality at Google Cloud, said in an interview. “It’s not a world where you have to wait for some hardware to come out, it’s not something you need to wait for people to buy. It’s already a reality, and we can make these experiences available to all the billions of people who already have a smartphone.”

The transformation of augmented reality from technological curiosity to mainstream marketing tool is well underway. An important development in that journey has been Immersive Stream for XR. With this new service from Google Cloud, the rendering of 3D and AR experiences is no longer dependent on the limited hardware of the smartphone; instead, it moves it to the cloud, where massive numbers of high-performance servers can bring creators’ visions to life, all in the palm of your hand. 

Immersive Stream for XR effectively enables any device to become an extended reality device. Interactive experiences can suddenly be streamed to devices as quickly and simply as we have long seen with YouTube videos or high-definition games, allowing users to interact right away, all without having to download an app. 

“We’re seeing the ability of technology coming into play to expand these immersive experiences,” Brian Kracik, head of telecom industry marketing at Google Cloud, said. “We want companies to be able to create these more interactive, photorealistic opportunities for their customers.”

Freeing extended reality for the mainstream

The time when augmented reality was dominated by pixelated digital monsters floating in space has come to an end. Services like Immersive Stream for XR can finally allow for rendering of photorealistic 3D digital objects and spaces, opening up a massive range of possibilities and opportunities for what can be done.

A number of important technical advances have made this moment possible.

Mobile devices today are generally not capable of handling graphics-intensive, photorealistic interactive experiences. Immersive Stream for XR solves this problem by taking the processing of 3D digital objects away from the graphics processing units of the phone and moving them to processors in the cloud, then streaming to the phone in the same way a video would stream to the device. 

To simplify things on the creative end, developers can use industry standard tools such as Unreal Engine to create these experiences. And, with the advances in mobile connectivity and 5G wireless networks, most phones today have access to the high throughput and low latency connections needed to make possible the streaming of 3D digital images into physical space. 

This connectivity is core to the responsiveness now available to creators and consumers. When users transition from 3D to AR mode, for example, user actions, such as phone position, user taps, and swipes, are sent back to the service to facilitate interactivity within the augmented world. Immersive Stream for XR is able to display the digital objects in realistic detail in AR, even using ambient lighting from the user’s location to make it look natural.

Because of the tight interaction between the mobile device and cloud to create the AR experience, network latency between the user and where the object is being rendered can be a concern. With the power of robust data centers and connective networks like those that make up Google Cloud, companies and developers can render their experiences as close as possible to the user, minimizing network latency. 

Additionally, Google has developed unique device-side reprojection algorithms that intelligently predict what 3D objects look like from novel angles, helping compensate for any latency that can’t be eliminated, due to distance and physics.

The art of the possible with immersive streaming

Augmented reality captured the public imagination when Pokemon Go — a game where people use their phone to hunt 3D digital characters in the real world — was released in 2016. To this point, when many people talk about augmented or extended reality, Pokemon Go is often their first and only reference.

Brands now have a chance to change that perception of augmented reality by providing easy to use extended reality experiences straight to people’s smartphone, without any special apps or large downloads.

“It’s not just for gaming anymore,” Kracik said. “We’re making it easier for consumers to engage with these products. It’s QR code driven, with nothing to download to a mobile device. I can just stream the experience. By using the processing power of the cloud, instead of the devices, we not only preserve the battery, we make development and use simple, easy, and engaging. It’s to the point where you can find so much detail, it’s crazy.”

Automotive manufacturers have been using extended reality experiences for several years, particularly since the pandemic, to give potential buyers detailed visuals of vehicles without the need of going to a dealer, or even leaving their couch. For instance, you can experience the BMW i4 and iX in realistic detail through your phone, even projecting it into your driveway or garage to get a closer experience of ownership.

Or you could remodel your kitchen before you even go shopping by using Lowe’s Immersive Stream experience to see how different colors or products will look once installed.

Fashion, tourism, and education are among the industries that have benefited so far from presenting consumers XR experiences on their phones. For instance, KDDI allows people to virtually try on clothes to help enable sustainable manufacturing. The Singapore Tourism Board is prototyping next-generation travel planning experiences, providing cinematic discovery of Singapore wherever you are in the world.

“There has been a general shift in preference from things we’d go and do in person — like buying a car, going retail shopping, or meeting with each other — that are now taking place virtually,” Ochandarena said.

Augmenting the world

A particularly intriguing opportunity with streaming XR experiences from the cloud is the ability to add multiple data sources. Instead of just rendering a single object like a car in the driveway or a virtual tour guide, the experience can be given depth from several perspectives.

For instance, think about sports. Most modern stadiums have many sensors to track the play of the game, whether it's the trajectory of a home run ball or a deep football pass, or how a soccer player positioned their body for a tricky header. Now what if a fan could “see” what the sensor sees on their device, getting multiple perspectives and real-time stats. And athletes and coaches could potentially tap this data, too.

“Imagine being able to capture what is going on during a sporting event and then give end users a more immersive way of experiencing it,” Ochandarena said. “Users can put themselves in the eyes of their favorite player or position, or even ‘be the ball.’ It’s all possible.”

Extrapolating further, real-world augmentation can be done nearly anywhere with a data source and a quality internet connection. 

One interesting example was presented by Google Maps, when it announced Immersive View at Google I/O 2022. With Immersive View and Immersive Stream for XR, enterprises will be able to tap into the power of Google Cloud and Maps to enable richer augmented experiences of almost anywhere on the planet on nearly any device.

It could someday evolve to become a digital twin for the entire planet.

In the meantime, a new wave of companies will emerge that use immersive technology to solve specific use cases in industries like real estate, education, or healthcare. A commercial real estate firm could visually simulate how a new skyscraper might change the look of a neighborhood before it’s built. In education, students in the future could learn complex topics like biology by interacting with 3D models of molecules, learn history by virtually touring landmarks across the world, or develop interpersonal skills by chatting face to face with avatars powered by conversational AI.

 Extended reality will soon be a mainstream opportunity to increase engagement with consumers, fans, and the public at large. Organizations are just starting to tap into the potential of what will be possible.“The cloud is the platform,” Kracik said. “This becomes a new tool for companies to get out in front of people in a new way and create more engagement.”