Infinite Nature and the nature of industries: This 'wild' demo shows the diverse possibilities of AI

Imagine exploring the boundless wonders of nature with a team of the world’s most knowledgeable biologists, zoologists, and ecologists. They could walk you through any environment and draw connections some of their peers in the field might not even make.

This is the world of Infinite Nature.

Introduced at Google Cloud Next ‘24, Infinite Nature was an experiment to showcase some of the more unexpected capabilities latent within generative AI. Infinite Nature is an immersive experience that harnesses the features and insights of our latest generation Gemini models to reveal surprising links within the natural world. (We’ve also brought the demo to I/O this week — come check it out at the Demo Porch at the Shoreline if you'll be attending.)

Our inspiration was the endless diversity and ingenuity found across the Earth. Infinite Nature is an invitation to embark on a journey of discovery, guided by the powerful insights of Gemini.

Infinite Nature is also a showcase for those emergent AI capabilities. The same way we wanted to push the boundaries of imagination, and inspire the many visitors to Infinite Nature to reexamine their world and the beautiful, serendipitous connections across it, we hoped to inspire a similar spirit of exploration in these audiences. What could the same models bring to their logistics pinch points, their financial transactions, their marketing collateral?

While it might at first seem odd to think about biodiversity as a kind of data, when it comes to Infinite Nature, that’s exactly what it is, with biomarkers and metadata serving as the connection points that bring a cardinal, maccaw, and southern red bishop together (prompt: “show me some red birds”). Now apply that same thinking to a query like “What were the top selling items in our 10 busiest stores in the Midwestern U.S.?”

Put another way, what do you want to do with your data that you never knew you could?

What is Infinite Nature?

At its core, Infinite Nature is a generative AI playground powered by Vertex AI and Gemini on Google Cloud. Here's how it works:

1. Ask a question: Start by posing a question or prompt about nature. This could be anything from "What are the most colorful birds?" to "Show me animals that live for a long time." The input for the experience can be audio, text, location coordinates, or images — which demonstrates the power of a natively multimodal model like Gemini that can easily understand a range of input types.
2. Multimodal AI analysis: The prompt is ingested by a tuned version of Gemini, which was trained on a 9 TB open dataset of multimedia wildlife data from the Global Biodiversity Information Facility (GBIF). Infinite Nature's uses LLM-powered search to find the most semantically relevant results from millions of real animal images and data points.
3. Generative text and visuals: Infinite Nature presents its findings in visually engaging clusters that are generated dynamically with Gemini. These clusters draw links between seemingly disparate concepts, sparking curiosity and inviting further exploration of the GBIF dataset.

The forest and the trees: Building Infinite Nature

Infinite Nature showcases the remarkable capabilities of AI in processing and connecting real-world datasets in real-time:

1. Harnessing the power of GBIF with multimodal semantic search: Infinite Nature leverages the GBIF dataset, which contain 7.3 million images animals worldwide, along with relevant metadata such as the species, diet, habitat and habits, longevity, and numerous other markers that can help the Infinite Nature model draw connections between these creatures.

This rich dataset was used to generate embeddings for each animal, a numerical vector that the model can use to understand and compare each datapoint; these were created using Vertex AI Multimodal Embeddings and multimodal semantic search with Vertex AI Vector Search.

These embeddings enable the system to grasp the nuanced intent behind each user’s prompt, so the model can identify the most semantically relevant results from millions of real animal images and traits From this intention, the model can then gather relevant images, even for complex queries like "hippos bathing in water" or "birds flying in a circle." All this takes place in a matter of milliseconds.

2. Generative recommendation for endless inspiration: Powered by Gemini's reasoning capabilities, Generative Recommendation (GR) dynamically generates poetic, evocative semantic queries based on your interests and its vast knowledge on the animals worldwide. For example, given a query like "hummingbird," it might produce queries like "A photo of a hummingbird clearwing perched on a flower, showcasing its translucent wings and colorful body." guiding you to discover fascinating aspects of the natural world.

3. Retrieval Augmented Generation (RAG) with Wikipedia: When you express interest in a specific animal, Infinite Nature employs retrieval-augmented generation, or RAG, to provide rich, in-depth information that’s grounded in sources outside the model in addition to its internal knowledge.

Once queried, Infinite Nature cross-referenced relevant data from 19,000 pages of Wikipedia for the animal species and 7.3 million rows of metadata in the GBIF dataset with Vertex AI Feature Store in milliseconds, and then feeds this context to Gemini. The LLM then generates insightful questions, answers, and visual descriptions related to the animal with largely reduced risk of the hallucination.

4. Function Calling for Natural Interaction: Gemini models support function calling that allows the system to gracefully map the results from the LLM to structured JSON outputs (a simplified data interchange format). This transfer eliminates the prompting and coding trial-and-error previously common when integrating an LLM with the system around it.

5. Powered by Google Cloud: Infinite Nature leverages various Google Cloud tools to bring the experience to life. BigQuery enables efficient data management, Vertex AI powers model training and serving, and other services ensure smooth performance and scalability.

The Magic of Discovery

Infinite Nature is more than just a collection of facts. It's a tool for sparking wonder. During Next '24, users were amazed at the breadth of connections the AI uncovered.

For example, inspired in the context of Las Vegas, one attendee asked, "What are the most iconic animals from Nevada?" The resulting clusters spanned topics from animals that thrive in the desert, to specifically covering bighorn sheep and the mountain bluebird, which is the state bird of the Sagebrush State.

The user interface and experience of Infinite Nature was specifically designed to evoke a sense of wonder and discovery. Dynamic visualizations offer a glimpse into the connections the AI uncovers, while a chat-like interface creates a sense of conversation. Interacting with Infinite Nature feels like embarking on a journey of discovery with a knowledgeable guide.

Throughout, our goal was to make the AI's "thought process" visible and transparent, as a way to increase the awareness of our audiences and bridge the ideas around their own use cases. A persistent on-screen module explains the reasoning behind each cluster of animals, enhancing the experience's transparency and encouraging deeper engagement.

Infinite Possibilities

The AI technologies behind Infinite Nature have the potential to revolutionize how companies interact with their customers and data across industries and uses. By harnessing the power of conversational AI, generative recommendation, and real-time multimodal semantic search and data retrieval, businesses can create experiences that are more engaging, intuitive, and impactful than ever before.

Imagine using a similar AI-driven approach to:

• Discover hidden patterns and connections in historical data
• Identify customer behavior in your platform
• Customize shopping paths for customers on your website
• Create new content for your brand grounded on your existing library

Infinite Nature offers a glimpse into the future of how we'll interact with information. It moves away from the traditional "question-and-answer" model of search engines, instead turning the focus towards exploration and discovery. AI doesn't just provide facts—it actively generates connections within vast datasets, sparking new insights and creative leaps that the user might never have considered.

This approach aligns with the way our own minds explore, learn, and find creative solutions. Infinite Nature shows us that AI can be a powerful thought partner. It hints at a future where interacting with information is more intuitive, personalized, and focused on unlocking the hidden potential within the world's knowledge.