3co reinvents the digital shopping experience with augmented reality on Google Cloud

 Giving people as close to a “try-before-you-buy” experience is essential for retailers. With the move to online shopping further accelerated by the COVID-19 pandemic, many people are now comfortable shopping online for items they previously only considered buying in stores. The problem for shoppers is that it still can be difficult to get what feels like more hands-on experiences of items given limitations with even some of today’s most advanced augmented reality (AR) technologies. And while retailers continue to invest heavily in creating the most life-like digital experiences possible, the results often come up short for shoppers with more digital buying options than ever. 

To make AR experiences more convincing for shoppers—and for anyone wanting richer, more immersive experiences in entertainment and other industries—the depiction of real-world physical objects in digital spaces needs to continue to improve and evolve. As avid plant lovers, we knew the experience of viewing and buying plants online was severely lacking. That prompted our initial exploration into rethinking what’s possible with AR: we built a direct-to-consumer app for buying plants in AR. However, during our time in the Techstars program, we quickly realized that improving how people see and experience plants online was just a fraction of a much bigger, multi-billion-dollar opportunity for us. Since 2018, 3co has been laser-focused (quite literally) on scaling 3D tech for all of e-commerce.

Closing the gap between imagination and reality with Google Cloud

With that in mind, 3co began developing breakthroughs needed in 3D computer vision. Our advanced artificial intelligence (AI) stack is designed to give companies an all-in-one 3D commerce platform to easily and cost-effectively create realistic 3D models of physical objects and stage them in virtual showrooms.

When building our AR platform, we quickly understood that engineering 3D simulations with sub-perceptual precision requires an enormous amount of compute power. Fortunately the problems are parallelizable. But it simply isn’t possible to 3D model the complex real world with superhuman precision on conventional laptops or desktops.

As a part of the Google for Startups Cloud Program, Startup Success Managers helped 3co plug into the full power of Google’s industry-leading compute capabilities. For several projects, we selected a scalable Compute Engine powerful enough to solve even the most complex 3D graphics optimizations at scale. Today with the A2 virtual machine, 3co leverages NVIDIA Ampere A100 Tensor Core GPUs to create more life-like 3D renderings over ten times faster. And this is just the beginning.

We’re also proud to have deployed a customized streaming GUI on top of Google’s monstrous machines, which allowed our colleagues across the world (including in Amsterdam and Miami) to plug-and-play with the latest 3D models on a world-class industrial GPU. I would highly recommend to companies solving super hard AI and/or 3D challenges in a distributed team to consider adopting cloud resources in the same way. It was a delight to see Blender render gigabyte 3D models faster than ever before in my life.

Equally critical, with our technology, 3D artists in retail, media and entertainment, and other industries pressured to deliver more—and more immersive AR—experiences can reduce costs and speed to generate photorealistic 3D models, as much as tenfold. We know this from our own work because we’ve seen computing costs to generate the highest-quality 3D experiences drop significantly—even though we run an advanced Compute Engine loaded with a powerful GPUs, high-end CPUs, and massive amounts of RAM. If the goal is to scale industry-leading compute power quickly for a global customer base, Google Cloud is the proper solution. 

Cloud Storage is another key but often overlooked component of the Google Cloud ecosystem, critical for 3co. We need the high throughput, low latency, and instant scalability delivered by local cloud SSDs to support the massive amounts of data we generate, store, and stream. The local SSDs complement our A2 compute engines and are physically attached to the servers hosting the virtual machine instances. This local configuration supports extremely high input/output operations per second (IOPS) with very low latency compared to persistent disks.

To top it off, Cloud Logging delivers us real-time log management at exabyte scale — ingesting analytic events that are streamed to data lakes with Pub/Sub – so we can know while enjoying the beach here in Miami, Florida that everything is going smoothly in the cloud.

Building the 3co AI stack with TensorFlow

Building one of the world's most advanced 3D computer vision solutions would not have been possible without TensorFlow and its comprehensive ecosystem of tools, libraries, and community resources. Since the launch of TensorFlow in 2015, I’ve personally built dozens of deep learning systems using this battle-hardened technology, an open source Google API for AI. Through TensorFlow on Google Cloud, 3co is able to scale its compute power for creation of truly photorealistic digital models of physical objects — down to microscopic computation of material textures, and deep representations of surface light transport from all angles.  

In several ways, TensorFlow is the go-to solution both for prototyping and for large-scale deployment of AI in general. Under-the-hood, TensorFlow uses a sophisticated compiler (XLA) for optimizing how computations are allocated on underlying hardware.

Unlike its competitors (e.g. PyTorch, JAX), TensorFlow can also compile binaries to run on TPUs (i.e. TFLite) and across device architectures (e.g. iOS, Android, JavaScript). This ability is important because 3co is committed to delivering 3D computer vision wherever it is needed, with maximum speed and accuracy. Through TensorFlow on Google Cloud, 3co has been able to speed up experimental validation of patent-pending 3D computer vision systems that can run the same TensorFlow code across smartphones, LIDAR scanners, AR glasses, and so much more.

TensorFlow also enables 3co’s neural networks to train faster, through an easy API for distributed training across many computers. Distributed deep learning was the focus of my masters thesis in 2013 (inspired by work from Jeff Dean, Andrew Ng, and Google Brain), so you can imagine how excited I was to see Google optimize these industry-leading capabilities for the open source community, over the following years. Parallelization of deep learning has consistently proven essential for creating this advanced AI, and 3co is no exception to this rule. As well, with faster AI training means faster conclusion of R&D experiments. As Sam Altman says, “The number one predictor of success for a very young startup: rate of iteration”. From day one, TensorFlow was built to speed up Google’s AI computing challenges at the biggest scale, but it also “just works” at the earliest stages of exploration. Through TensorFlow on Google Cloud, 3co is steadily improving our capabilities for autonomous photorealistic 3D modeling. Simple and flexible architectures for fast experimentation enable us to quickly move from concept to code, from code to state-of-the-art deployed ML models. Thus, Google has given 3co through TensorFlow a powerful tool needed to better serve customers with their modern AI and computer vision. 

In the future, 3co has big plans involving supercomputers of Google Cloud Tensor Processing Units (TPUs), so we plan to achieve even greater speed and cost optimization. Running TensorFlow on Cloud TPUs requires just a little bit of extra work by the AI developer, but Google is increasingly making it easier to plug-and-play on these gargantuan computing architectures. They truly are world class servers for AI. I remember being as excited as a little boy in a candy store, reading research back in 2017 on Google’s TPUs, which was the climax of R&D for literally dozens of super smart computer engineers. Since then, several versions of TPUs have been deployed internally at Google for many kinds of applications (e.g. Google Translate), and increasingly have been made more useful and accessible. Startups like 3co – and our customers – can benefit so much here. Through the use of advanced computer processors like TPUs, 3co expects to parallelize its AI to perform photorealistic 3D modeling of real scenes in real-time. Imagine the possibilities for commerce, gaming, entertainment, design, and architecture that this ability could unlock. 

Scaling 3D commerce with Google Cloud and credits

3co’s participation in the Google for Startups Cloud Program (facilitated via Techstars, we also can’t thank them enough) has been instrumental to our success in closing the gap between imagination and reality. It’s a mission we’ve been working on for years – and will continue to hone for many years to come. And this success is thanks to the Google for Startups Success team: they are truly amazing. They just care about you. If you’re a startup founder, just reach out to them: they really do wonders. We especially want to highlight the Google Cloud research credits which provided 3co access to vastly greater amounts of compute power. We are so grateful to Google Cloud for enabling 3co to scale its 3D computer vision services to customers worldwide. I love that 3co is empowered by Google to help many people see the world in a new light.  

If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.