Your ultimate guide to the latest in generative AI on Vertex AI

The world of generative AI is evolving at a pace that's nothing short of mind-blowing. It feels like just yesterday we were marveling at AI-generated images, and now we're having full-fledged conversations with AI chatbots that can write code, craft poetry, and even serve our customers (check out our list of 101 real-world gen AI use cases from the world's leading organizations).

The pace of innovation can be hard to keep up with — in 2023 we introduced over 500 new features in Vertex AI and we’re not slowing down this year. We put this blog together to help you keep track of the biggest announcements and make sense of what they mean for your business. We’ll keep it updated as new announcements come out, so bookmark this link and check out our new video series below also covering new announcements. 

Catch up on the latest announcements

As part of our Gemini at Work global event, we showcased nearly 50 new customer stories from organizations around the world to highlight just how impactful generative AI can be when you put it to work at scale. 

We’re inspired by what customers are building and excited to announce a few exciting updates to our models and platform, designed to help move gen AI from experimentation into production with Vertex AI.

Updates to Gemini Models

• What it is: The newly updated versions of Gemini 1.5 Pro and Flash models, both GA, deliver quality improvements in math, long context understanding, and vision.

• Why it matters: Our objective is to bring you the best models suited for enterprise use cases, by pushing the boundaries across performance, latency, and costs. From a latency standpoint, the new version of Gemini 1.5 Flash is nearly 2.5x faster than GPT-4o mini. 

• Get started: Access Gemini 1.5 Pro and Flash in the Google Cloud console. 

Reduced Gemini 1.5 Pro Pricing 

• What it is: We are reducing costs of Gemini 1.5 Pro by 50% across both input and output tokens, effective on Vertex AI on October 7, 2024. 

• Why it matters: We are committed to making AI accessible for every enterprise.  In August, we improved Gemini 1.5 Flash to reduce costs by up to 80% (see below). These world class models can be coupled with capabilities like context caching to even further reduce the cost and latency of your long context queries. Using Batch API instead of standard requests can further optimize costs for latency insensitive tasks. 

• Get started: Visit the pricing page to learn more. 

Updates to Imagen 3

• What it is: Google’s latest image generation model, delivering outstanding image quality, multi-language support, built-in safety features like Google DeepMind’s SynthID digital watermarking, and support for multiple aspect ratios.

• Why it matters: There are several improvements over Imagen 2 — including over 40% faster generation for rapid prototyping and iteration; better prompt understanding and instruction-following; photo-realistic generations, including of groups of people; and greater control over text rendering within an image. 

• Get started: Apply for access to Imagen 3 on Vertex AI. 

Controlled generation is now GA

• What it is: Controlled generation lets customers define Gemini model outputs according to specific formats or schemas. 

• Why it matters: Most models cannot guarantee the format and syntax of their outputs, even with specified instructions. Vertex AI controlled generation lets customers choose the desired output format via pre-built options like JSON and ENUM, or by defining custom formats. 

• Get started: Visit documentation to learn more.  

Batch API

• What it is: Batch API (currently in preview) is a super-efficient way to send large numbers of non-latency sensitive text prompt requests, supporting use cases such as classification and sentiment analysis, data extraction, and description generation. 

• Why it matters: It helps speed up developer workflows and reduces costs by enabling multiple prompts to be sent to models in a single request.

• Get started: View documentation to get started.

Supervised fine tuning (SFT) for Gemini 1.5 Flash and Pro

• What it is: SFT for Gemini 1.5 Flash and Pro is now generally available. SFT adapts model behavior with a labeled dataset, adjusting the model's weights to minimize the difference between its predictions and the actual labels. 

• Why it matters: SFT allows you to tune the model to be more precise for your enterprise task. It's particularly effective for domain-specific applications where the language or content significantly differs from the data the large model was originally trained on. 

• Get started: Visit documentation to learn more. 

Distillation techniques in Vertex AI

• What it is: Train smaller, specialized models that inherit the knowledge of the larger Gemini model, achieving comparable performance with the flexibility of self-hosting your custom model on Vertex AI. 

• Why it matters: Deploying large language models can be a resource-intensive challenge. With distillation techniques in Vertex AI, you can leverage the power of those large models while keeping your deployments lean and efficient. 

• Get started: Fill out this form for early access.

Prompt Optimizer, now in preview  

• What it is: Based on Google Research’s publication on automatic prompt optimization (APO) methods, Prompt Optimizer adapts your prompts using the optimal instructions and examples to elicit the best performance from your chosen model. 

• Why it matters: Vertex AI’s Prompt Optimizer helps you avoid the tedious trial-and-error of prompt engineering. Plus, our prompt strategies guide helps you make the models more verbose and conversational.

• Get started: Learn more in documentation.

Prompt Management SDK

• What it is: Vertex AI’s Prompt Management SDK allows users to retrieve and organize prompts. It lets you version prompts, restore old prompts, and generate suggestions to improve performance. 

• Why it matters: This makes it easier for you to get the best performance from gen AI models at scale, and to iterate more quickly from experimentation to production. 

• Get started: The prompt management SDK will be generally available in the coming weeks.

Multimodal function calling

• What it is: Function calling is a built-in feature of the Gemini API that translates natural language into structured data and back. 

• Why it matters: Now, with multimodal function calling, your agents can also execute functions where your user can provide images, along with text, to help the model pick the right function and function parameters to call.

• Get started: Learn more in documentation.

Org policy for models in Model Garden

• What it is: You can now govern the models available in Model Garden. This includes limiting access to gen AI features, to only specific vetted models, or to tuning and other advanced capabilities. Org policies can be applied to all models in Model Garden as well as those imported from Hugging Face through Model Garden. These policies can be set on an organization, folder or project resource to enforce the constraint on that resource and any child resources.

• Why it matters: The ability to control access to the models made available through the Model Garden has been a top priority for many customers looking for governance and control tooling. Org Policies allows granular access control of models and enhances Google Cloud’s enterprise readiness in an age of many models from many providers.

• Get started: Learn more in documentation.

Previous Announcements

Best models from Google and the industry

We’re committed to providing the best model for enterprises to use - Vertex AI Model Garden provides access to 150+ models from Google, Partners and the open community so customers can select the model for the right price, performance, and latency considerations.

No matter what foundation model you use, it comes with enterprise ready tooling and integration to our end to end platform. 

Gemini 1.5 Flash is GA

• What it is: Gemini 1.5 Flash combines low latency, highly competitive pricing, and our 1 million-token context window.

• Why it matters: Gemini 1.5 Flash is an excellent option for a wide variety of use cases at scale, from retail chat agents, to document processing, to research agents that can synthesize entire repositories.

• Get started: Click here to get started now with Gemini 1.5 Flash on Vertex AI. 

Lower pricing for Gemini 1.5 Flash

• What it is: We've updated Gemini 1.5 Flash to reduce the input costs by up to ~85% and output costs by up to ~80%, starting August 12th, 2024. 

• Why it matters: This is a big price drop on Gemini Flash, a world-class model with a 1 million context window and multi-modal inputs. Plus, coupled with capabilities like context caching you can significantly reduce the cost and latency of your long context queries. Using Batch API instead of standard requests can further optimize costs for latency insensitive tasks. 

• Get started: View pricing to learn more and try out Gemini 1.5 Flash today.

Gemini 1.5 Pro, GA with 2-million -token context capabilities 

• What it is: Now available with an industry-leading context window of up to 2 million tokens, Gemini 1.5 Pro is equipped to unlock unique multimodal use cases that no other model can handle.

• Why it matters: Processing just six minutes of video requires over 100,000 tokens and large code bases can exceed 1 million tokens — so whether the use case involves finding bugs across countless lines of code, locating the right information across libraries of research, or analyzing hours of audio or video, Gemini 1.5 Pro’s expanded context window is helping organizations break new ground. 

• Get started: Click here to get started now 

More Languages for Gemini

• What it is: We're enabling Gemini 1.5 Flash and Gemini 1.5 Pro to understand and respond in 100+ languages. 

• Why it matters: We’re making it easier for our global community to prompt and receive responses in their native languages.

• Get started: View documentation to learn more.

Gemma 2

• What it is: The next generation in Google’s family of open models built to give developers and researchers the ability to share and commercialize their innovations, using the same technologies used to create Gemini. 

• Why it matters: Available in both 9-billion (9B) and 27-billion (27B) parameter sizes, Gemma 2 is much more powerful and efficient than the first generation, with significant safety advancements built in. 

• Get started: Access Gemma 2 on Vertex AI here.

Meta’s Llama 3.1 

• What it is: Llama 3.1 models are now available on Vertex AI as a pay as you go API, this includes 405B, 70B and 8B (coming in early September). 

• Why it matters: 405B is the largest openly available foundation model to date. 8B and 70B are also new versions that excel at understanding language nuances, grasping context, and performing complex tasks such as translation and dialogue generation. You can access the new models in just a few clicks using Model-as-a-Service, without any setup or infrastructure hassles. 

• Get started: To access Llama 3.1, visit Model Garden

Mistral AI’s latest models

• What it is: We added Mistral Large 2, Nemo and Codestral (Google Cloud is the first hyperscaler to introduce Codestral). 

• Why it matters: Mistral Large 2 is their flagship model which offers their best performance and versatility to date and Mistral Nemo is a 12B model that delivers exceptional performance at a fraction of the cost. Codestral is Mistral AI’s first open-weight generative AI model explicitly designed for code generation tasks. You can access the new models in just a few clicks using Model-as-a-Service, without any setup or infrastructure hassles. 

• Get started: To access the Mistral AI models, visit Model Garden (Codestral, Large 2, Nemo) or check out the documentation

Jamba 1.5 Model Family from AI21 Labs

• What it is: Jamba 1.5 Model Family  — AI21 Labs’ new family of open models — is in public preview on Vertex AI Model Garden, including:  

• Jamba 1.5 Mini: AI21’s most efficient and lightweight model, engineered for speed and efficiency in tasks including customer support, document summarization, and text generation.

• Jamba 1.5 Large: AI21’s most advanced and largest model that can handle advanced reasoning tasks — such as financial analysis — with exceptional speed and efficiency. 

• Why it matters: AI21’s new models join over 150 models already available on Vertex AI Model Garden, further expanding your choice and flexibility to choose the best models for your needs and budget, and to keep pace with the continued rapid pace of innovation. 

• Get started: Select the Jamba 1.5 Mini or Jamba 1.5 Large model tile in Vertex AI Model Garden. 

Anthropic’s Claude 3.5 Sonnet

• What it is: We recently added Anthropic’s newly released model, Claude 3.5 Sonnet, to Vertex AI. This expands the set of Anthropic models we offer, including Claude 3 Opus, Claude 3 Sonnet, Claude 3 Haiku. You can access the new models in just a few clicks using Model-as-a-Service, without any setup or infrastructure hassles. 

• Why it matters: We're committed to empowering customer choice and innovation through our curated collection of first-party, open, and third-party models available on Vertex AI.

• Get started: Begin experimenting with or deploying in production Claude 3.5 Sonnet on Vertex AI.

End-to-end model building platform with choice at every level

Vertex AI Model Builder enables you to build or customize your own models, with all the capabilities you need to move from prototype to production.

Lower cost with context caching for both Gemini 1.5 Pro and Flash

• What it is: Context caching is a technique that involves storing previous parts of a conversation or interaction (the "context") in memory so that the model can refer back to it when generating new responses

• Why it matters: As context length increases, it can be expensive and slow to get responses for long-context applications, making it difficult to deploy to production. Vertex AI context caching helps customers significantly reduce input costs, by 75 percent, leveraging cached data of frequently-used context. Today, Google is the only provider to offer a context caching API. 

• Get started: Learn more in documentation.

New model monitoring capabilities

• What it is: The new Vertex AI Model Monitoring includes

• Support for models hosted outside of Vertex AI (e.g. GKE, Cloud Run, even multi-cloud & hybrid-cloud)

• Unified monitoring job management for both online and batch prediction

• Simplified configuration and metrics visualization attached to the model, not the endpoint

• Why it matters: Vertex AI’s new model monitoring features provide a more flexible, extensible, and consistent monitoring solution for models deployed on any serving infrastructure (even outside of Vertex AI, e.g. Google Kubernetes Engine, Cloud Run, Google Compute Engine and more).

• Get started: Learn more in this blog.

Ray on Vertex AI is GA

• What it is: Ray provides a comprehensive and easy-to-use Python distributed framework. With Ray, you configure a scalable cluster of computational resources and utilize a collection of domain-specific libraries to efficiently distribute common AI/ML tasks like training, serving, and tuning. 

• Why it matters: This integration empowers AI developers to effortlessly scale their AI workloads on Vertex AI's versatile infrastructure, which unlocks the full potential of machine learning, data processing, and distributed computing.

• Get started: Ready the blog to learn more.

Prompt Management

• What it is: Vertex AI Prompt Management, now in preview, provides a library of prompts for use among teams, including versioning, the option to restore old prompts, and AI-generated suggestions to improve prompt performance. 

• Why it matters: This feature makes it easier for organizations to get the best performance from gen AI models at scale, and to iterate more quickly from experimentation to production. Customers can compare prompt iterations side by side to assess how small changes impact outputs, and the service offers features like notes and tagging to boost collaboration. 

• Get started: Visit documentation to learn more.

Gen AI Evaluation Services 

• What it is: We now support Gen AI Evaluation in GA to help users evaluate any generative model or application. While leaderboards and reports offer insights into overall model performance, they don't reveal how a model handles your specific needs. The Gen AI Evaluation Service helps you define your own evaluation criteria, ensuring a clear understanding of how well generative AI models and applications align with your unique use case. Users can define their own evaluation metrics and also access our templates for metrics with various criteria (e.g. text quality, instruction following, fluency). 

• Why it matters: Evaluation is important at every step of your gen AI development process including model selection, prompt engineering, and model customization. Evaluating gen AI is integrated within Vertex AI to help you launch and reuse evaluations as needed.

• Get started: Learn more in documentation.

Develop and deploy agents faster, grounded in your enterprise truth

Vertex AI Agent Builder allows you to easily and quickly build and customize AI Agents - for any skill level. A core component of the Vertex AI Agent Builder is Vertex AI Search, enabling you to ground the models in your data or the web. 

Grounding at Vertex

You have many options for Grounding and RAG at Vertex. These capabilities address some of the most significant hurdles limiting the adoption of generative AI in the enterprise: the fact that models do not know information outside their training data, and the tendency of foundation models to “hallucinate,” or generate convincing yet factually inaccurate information. Retrieval Augmented Generation (RAG), a technique developed to mitigate these challenges, first “retrieves” facts about a question, then provides those facts to the model before it “generates” an answer – this is what we mean by grounding. Getting relevant facts quickly to augment a model's knowledge is ultimately a search problem.  

Read more at this blog post.

Grounding with Google Search is GA

• What it is: When customers select Grounding with Google Search for their Gemini model, Gemini will use Google Search, and generate an output that is grounded with the relevant internet search results. Grounding with Google Search also offers dynamic retrieval, a new capability to help customers balance quality with cost efficiency by intelligently selecting when to use Google Search results and when to use the model’s training data. 

• Why it matters: Grounding with Google Search is simple to use and makes the world’s knowledge available to Gemini.  Dynamic retrieval will save you money and will save your users time, only grounding when needed.

• Get started: Read documentation to learn more about how to get started.

Grounding with third-party datasets

• What it is: Vertex AI will offer a new service that lets customers ground their models and AI agents with specialized third-party data. We are working with providers like Moody's, MSCI, Thomson Reuters and Zoominfo to enable access to their datasets.

• Why it matters: These capabilities will help customers build AI agents and applications that offer more accurate and helpful responses. 

• Get started: Coming soon, contact sales to learn more. 

Grounding with high-fidelity mode 

• What it is: High-fidelity mode is powered by a version of Gemini 1.5 Flash that’s been fine-tuned to only use customer-provided content to generate answers and ensures high levels of factuality in response. 

• Why it matters: In data-intensive industries like financial services, healthcare, and insurance, generative AI use cases often require the generated response to be sourced from only the provided context, not the model’s world knowledge. Grounding with high-fidelity, announced in experimental preview, is purpose-built to support such grounding use cases, including summarization across multiple documents, data extraction against a set corpus of financial data, or processing across a predefined set of documents.

• Get started: Contact sales to learn more. 

Expanding Vector Search to support hybrid search

• What it is: Vector Search, the ultra high performance vector database powering Vertex AI Search, DIY RAG, and other embedding use cases at global scale, now offers hybrid search in Public Preview. 

• Why it matters: Embeddings are numerical representations that capture semantic relationships across complex data (text, images, etc.). Embeddings power multiple use cases, including recommendation systems, ad serving, and semantic search for RAG. Hybrid search combines vector-based and keyword-based search techniques to ensure the most relevant and accurate responses for users. 

• Get started: Visit documentation to learn more about Vector Search.

LangChain on Vertex

• What it is: An agent development SDK and container runtime for LangChain. With LangChain on Vertex AI you can select the model you want to work with, define tools to access external APIs, structure the interface between the user and the system components in an orchestration framework, and deploy the framework to a managed runtime.

• Why it matters: LangChain on Vertex AI simplifies and speeds up deployment while being secure, private and scalable. 

• Get started: Visit documentation to learn more. 

Vertex AI extensions, function calling and ​​data connectors 

• What it is: 

• Vertex AI extensions are pre-built reusable modules to connect a foundation model to a specific API or tool. For example, our new code interpreter extension enables models to execute tasks that entail running Python code, such as data analysis, data visualization, and mathematical operations. 

• Vertex AI function calling enables a user to describe a set of functions or APIs and have Gemini models intelligently select, for a given query, the right API or function to call, along with the appropriate API parameters.

• Vertex AI data connectors help ingest data from enterprise and third-party applications like ServiceNow, Hadoop, and Salesforce, connecting generative applications to commonly-used enterprise systems.

• Why it matters: With these capabilities, Vertex AI Agent Builder makes it easy to augment grounding outputs and take action on your user’s behalf. 

• Get started: Visit documentation to learn more about Vertex AI extensions, function calling and ​​data connectors.

Firebase Genkit 

• What it is: Genkit is an open-source TypeScript/JavaScript and Go framework designed by Firebase to simplify the development, deployment, and monitoring of production-ready AI applications.

• Why it matters: With the Vertex AI plugin for Genkit, developers can now take advantage of Google models like Gemini and Imagen 2, as well as text embeddings. Additionally Vertex Eval Service is baked into the Genkit local development experience along with OpenTelemetry tracing.

• Get started: Learn more in documentation.

LlamaIndex on Vertex AI

• What it is: LlamaIndex on Vertex AI simplifies building your own search engine for retrieval-augmented generation (RAG), from data ingestion and transformation to embedding, indexing, retrieval, and generation.

• Why it matters: Vertex AI customers can leverage Google’s models and AI-optimized infrastructure alongside LlamaIndex’s simple, flexible, open-source data framework, to connect custom data sources to generative models.

• Get started: Visit documentation to learn more.

Built on a foundation of scale & enterprise readiness

The revolutionary nature of generative AI requires a platform that offers privacy, security, control, and compliance capabilities organizations can rely on. Google Cloud is committed to helping our customers leverage the full potential of generative AI with privacy, security, and compliance capabilities. Our goal is to build trust by protecting systems, enabling transparency, and offering flexible, always-available infrastructure, all while grounding efforts in our AI principles.

Dynamic Shared Quota

• What it is: With Dynamic Shared Quota, we offer increasing the quota limits for a model (online serving) to the maximum allowed per region. This way we limit the number of queries per second (QPS) that customers can run by the shared capacity of all the queries running on a Servo station (multi-region), instead of limiting a customer’s QPS by a quota. Dynamic Shared Quota is only applicable to Pay-as-you-go Online Serving. For customers that require a consistent or more predictable service level, including SLAs, we offer Provision Throughput.

• Why it matters: By dynamically distributing on-demand capacity among all queries being processed for Pay-as-you-go customers, Google Cloud has eliminated the need to submit quota increase requests (QIRs). Customers can still set a self-imposed quota called a consumer quota override to control cost and prevent budget overruns.

• Get started: Learn more in documentation.

Provisioned Throughput is GA 

• What it is: Provisioned throughput lets customers responsibly scale their usage of Google’s first-party models, like 1.5 Flash, providing assurances for both capacity and price. 

• Why it matters: This Vertex AI feature brings predictability and reliability to customer production workloads, giving them the assurance required to scale gen AI workloads aggressively.  We have also made it easier than ever for customers to set up PT via a Self Service flow. Customers can now estimate their needs and purchase Provisioned Throughput for Google’s 1P foundation models via the console, bringing the E2E experience down from weeks to minutes for pre-approved orders subject to available capacity and removing the need for manual order forms.

• Get started: Follow these steps to purchase a Provisioned Throughput subscription.

Data residency for data stored at-rest guarantees in more countries

• What it is: We have data residency for data stored at-rest guarantees in 23 countries (13 of which were added in 2024), with additional guarantees around limiting related ML processing to the US and EU. We are also working on expanding our ML processing commitments to eight more countries, starting with four countries in 2024.

• Why it matters: Customers, especially those from regulated industries, demand control over where their data is stored and processed when using generative AI capabilities. 

• Get started: Learn more here.

To keep up with all of the latest releases, don’t forget to check our Vertex AI release notes. 

All of these enhancements are a direct response to what you, our customers, have been asking for. We believe an enterprise AI platform is key to success in production and our goal is to not just build the best platform, but to provide an AI ecosystem that makes enterprise-scale AI accessible.

To learn about how Vertex AI can help you, contact us for a free consultation.