Vertex AI generative AI notebook tutorials

This document contains a list of notebook tutorials for Generative AI on Vertex AI. They're end-to-end tutorials that show you how to use some of the GenAI LLMs.

Featured tutorials

• Multimodal use cases with Gemini

  Explore various use cases with multimodal with Gemini.
  Colab | GitHub

• Function Calling with the Vertex AI Gemini API & Python SDK

  Use the Vertex AI Gemini API with the Vertex AI SDK for Python to make function calls using the Gemini 1.0 Pro (gemini-1.0-pro) model.
  Colab | GitHub

• Get Started with Grounding with Gemini in Vertex AI

  Use generative text models to generate content grounded in your documents and data.

  Colab | GitHub

List of tutorials

Filter by: Gemini BigQuery Bulk-question-answering Captioning Chain of thought Chat Chatbot Chirp Code Code-chat Code-completion Code-generation Codey Contract Conversation Data Loader Data store DataFrames Deploy Document Document AI Document-QA Document-summarization Embeddings Evaluation Filter Function-calling Generation Ideation Image Imagen Indexing LangChain Language Language prompts Language-model Metadata Multimodal Orchestration PaLM prompts Python SDK Question-answering React Recognition Retail Retrieval Retrieval-augmented-generation Search Similarity Speech Summarization Text Text-classification Text-embeddings Text-extraction Text-generation Tuning Vector Search Vertex AI Search Visual-question-answering Visualization

Gemini Multimodal	Introduction to Gemini 1.5 Pro (Preview) Use Gemini 1.5 Pro (Preview) to analyze audio files, understand video, extract information from a PDF, and process multiple types of media simultaneously.	View on GitHub
Gemini Multimodal	Analyze a codebase Use this notebook to learn how to generate code, summarize a codebase, debug, improve code, and assess code with Gemini 1.5 Pro (Preview).	View on GitHub
Gemini Multimodal	Get started with Gemini (cUrl) Use the Gemini API, which gives you access to Google's latest large language models, with REST/curl.	Open in Colab View on GitHub
Gemini Multimodal	Get started with Gemini (Python SDK) Use the Gemini API, which gives you access to Google's latest large language models, with the Vertex AI SDK for Python.	Open in Colab View on GitHub
Gemini Multimodal	Multimodal use cases with Gemini The Gemini model is a groundbreaking multimodal language model developed by Google AI, capable of extracting meaningful insights from a diverse array of data formats, including images, and video. This notebook explores various use cases with multimodal prompts.	Open in Colab View on GitHub
Gemini evaluation	Getting started with rapid evaluation in the Vertex AI SDK for Python Use rapid evaluation to evaluate the Gemini model on an evaluation task, with the Vertex AI SDK for Python.	Open in Colab View on GitHub
Gemini evaluation	Prompt Engineering, Evaluation, and Prompt Templating with Gemini Use rapid evaluation for prompt engineering and evaluation with the Gemini model, and use the prompt template for prompt design, with the Vertex AI SDK for Python.	Open in Colab View on GitHub
Gemini evaluation	Score and Select Generative Models Use rapid evaluation to score and evaluate different generative models on a specific evaluation task, then visualize and compare the evaluation results for the task, with the Vertex AI SDK for Python.	Open in Colab View on GitHub
Gemini evaluation	Evaluate and Select Gemini Model Generation Settings Use rapid evaluation to evaluate and select temperature and other model generation configurations of Gemini and compare the metric results of different generation settings on quality, fluency, safety, and verbosity, with the Vertex AI SDK for Python.	Open in Colab View on GitHub
Gemini evaluation	Define Your Evaluation Metrics with Gemini Use rapid evaluation to evaluate with custom-defined evaluation metrics, with the Vertex AI SDK for Python.	Open in Colab View on GitHub
Gemini evaluation	Evaluate the Gemini 1.0 Pro model function call quality Generate function calls with Gemini 1.0 Pro model, and use rapid evaluation to evaluate the Gemini 1.0 Pro model function call quality, with the Vertex AI SDK for Python.	Open in Colab View on GitHub
Gemini Multimodal Streamlit	Deploy a Streamlit app to Cloud Run with Gemini Pro Sample app to deploy a simple chatbot app using Streamlit to Cloud Run with Gemini Pro.	View on GitHub
Gemini Multimodal Retrieval-augmented-generation	Multimodal RAG Extending from RAG, which is traditionally performed over text data, this notebook shows how you can perform RAG over multimodal data to do Q&A in a scientific paper containing text and images.	Open in Colab View on GitHub
Gemini Multimodal	Using Gemini in Education Using the Gemini model in education, with various examples of prompts, and across modalities including images and video.	Open in Colab View on GitHub
Gemini Multimodal	Using Gemini for Multimodal Retail Recommendations In the world of retail, recommendations play a pivotal role in influencing customer decisions and driving sales. In this notebook, you will learn how to harness the power of multimodality to perform retail recommendations to help a customer choose the best chair among four images of chairs, based on an image of their living room.	Open in Colab View on GitHub
Gemini Multimodal Function-calling	Intro to Function Calling with Gemini Use the Gemini Pro model to: Generate function calls from a text prompt to get the weather for a given location Generate function calls from a text prompt and call an external API to geocode addresses Generate function calls from a chat prompt to help retail users	Open in Colab View on GitHub
Tuning	Tune foundational models with Vertex AI Walk through the entire setup and integration process. From environment setup, to foundational model selection, and then to tuning with Vertex AI.	Open in Colab View on GitHub
Evaluation	Vertex AI LLM Evaluation Services Use Vertex AI LLM Evaluation Services in conjunction with other Vertex AI services.	View on GitHub
LangChain	Run a Langchain Chain Run a LangChain chain and print out details of what's happening in each step of the chain and with optional debugging breakpoints.	Open in Colab View on GitHub
Text	Advanced Prompt Engineering Training Use Chain of Thought and ReAct (Reasoning + Acting) to engineer prompts and reduce hallucinations.	Open in Colab View on GitHub
Embeddings	Use Vertex AI Embeddings for Multimodal and Vector Search Create text-to-image embeddings using the DiffusionDB dataset and the Vertex AI Embeddings for Multimodal model. The embeddings are uploaded to the Vector Search service, which is a high scale, low latency solution to find similar vectors for a large corpus.	Open in Colab View on GitHub
Embeddings	Semantic Search using Embeddings Create an embedding generated from text and perform a semantic search. The embeddings are generated using Google ScaNN: Efficient Vector Similarity Search.	Open in Colab View on GitHub
Evaluation	AutoSxS: Evaluate an LLM in Vertex AI Model Registry against a third-party model Use Vertex AI automatic side by side (AutoSxS) to evaluate the performance between a generative AI model in Vertex AI model registry and a third-party language model.	Open in Colab View on GitHub
Evaluation	AutoSxS: Check autorater alignment against a human-preference dataset Use Vertex AI automatic side by side (AutoSxS) to determine how well the autorater aligns with human raters.	Open in Colab View on GitHub
Tuning	Vertex AI LLM Reinforcement Learning from Human Feedback Use Vertex AI RLHF to tune a large-language model (LLM). This workflow improves a model's accuracy by fine-tuning a base model with a training dataset.	Open in Colab View on GitHub
Tuning	Vertex AI Batch Inference with RLHF-tuned Models This tutorial demonstrates how to perform inference on RLHF-tuned OSS large-language models (LLMs) end-to-end with Vertex AI.	Open in Colab View on GitHub
Embeddings	Text Embedding API Try the new text embedding models.	Open in Colab View on GitHub
Tuning	Vertex AI Tuning a PEFT model Tune a PEFT large-language model (LLM) and make a prediction. This workflow improves a model's accuracy by fine-tuning a base model with a training dataset.	Open in Colab View on GitHub
Text	Use the Vertex AI SDK with Large Language Models Use the Vertex AI SDK to run Large Language Models on Vertex AI. Test, tune, and deploy generative AI language models. Get started by exploring examples of content summarization, sentiment analysis, chat, text embedding, and prompt tuning.	Open in Colab View on GitHub
Data store Indexing Search Conversation	Vertex AI Search and Conversation Data Store Status Checker Vertex AI Search and Conversation Data Store Status Checker is a notebook that uses the Cloud Discovery Engine API to check a Data Store for indexed documents. It allows the user to perform the following tasks: Check Indexing Status of given Data Store ID. List all documents in a given Data Store ID. List all indexed URLs for a given Data Store ID. Search all indexed URLs for a specific URL within a given Data Store ID.	Open in Colab Open in GitHub
Speech Recognition Chirp	Get started with Chirp on Google Cloud This notebook is an introduction to Chirp, a speech-to-text service that uses Google's state-of-the-art speech recognition technology. It provides a simple and easy-to-use interface for developers to build speech-enabled applications.	Open in Colab Open in GitHub
Filter Metadata Search	Vertex AI Search with Filters & Metadata Vertex AI Search is a fully managed service that lets you build and deploy search applications on Google Cloud. This notebook shows how to use filters and metadata in search requests to Vertex AI Search.	Open in Colab Open in GitHub
Document Question-answering Retrieval-augmented-generation	Document Q&A With Retrieval Augmented Generation This notebook demonstrates how to use Retrieval Augmented Generation (RAG) to build a question answering system for Google documentation. It shows how to use RAG to generate text that answers a given question, and how to use RAG to improve the performance of a question answering system.	Open in Colab Open in GitHub
Contract Generation Retrieval Search	Retrieval Augmented Generation (Using Open Source Vector Store) - Procurement Contract Analyst - Palm2 & LangChain This notebook demonstrates how to use retrieval augmented generation to generate contract text. It uses the Palm2 & LangChain models, which are pre-trained on large corpora of legal and financial text.	Open in Colab Open in GitHub
Question-answering Retrieval-augmented-generation Search LangChain	Question Answering Over Documents This notebook shows how to ask and answer questions about your data by combining a Vertex AI Search engine with LLMs. In particular, we focus on querying 'unstructured' data such as PDFs and HTML files. In order to run this notebook you must have created an unstructued search engine and ingested PDF or HTML documents into it.	Open in Colab Open in GitHub
Bulk-question-answering Vertex AI Search Question-answering Search	Bulk Question Answering with Vertex AI Search This notebook shows how to answer questions from a CSV using a Vertex AI Search data store. It can be run in Colab or Vertex AI Workbench.	Open in Colab Open in GitHub
Language Orchestration LangChain PaLM	Getting Started with LangChain 🦜️🔗 + PaLM API This notebook provides an introduction to LangChain, a language model orchestration framework. It shows how to use LangChain with the PaLM API to create and deploy a text-to-text generation model.	Open in Colab Open in GitHub
BigQuery Data Loader LangChain	How to use the LangChain 🦜️🔗 BigQuery Data Loader This notebook demonstrates how to use the LangChain BigQuery Data Loader to load data from BigQuery into a LangChain model. The notebook provides step-by-step instructions on how to set up the data loader, load data into the model, and train the model.	Open in Colab Open in GitHub
Code Code-generation Retrieval-augmented-generation Codey	Use Retrieval Augmented Generation (RAG) with Codey APIs This notebook demonstrates how to use Retrieval Augmented Generation (RAG) with Codey APIs. RAG is a technique that combines code retrieval with code generation to produce more accurate and informative code completion suggestions.	Open in Colab Open in GitHub
Codey Code-generation Language	Getting Started with the Vertex AI Codey APIs - Code Generation This notebook provides an introduction to the Vertex AI Codey APIs for code generation. It covers the basics of how to use the APIs, including how to create and deploy code generation models, and how to use them to generate code.	Open in Colab Open in GitHub
Codey Code-completion Code-generation	Getting Started with the Vertex AI Codey APIs - Code Completion This notebook demonstrates how to use the Vertex AI Codey APIs to get code completion suggestions for your Python code. It also shows how to use the APIs to generate code snippets and run code snippets in a remote environment.	Open in Colab Open in GitHub
Codey Code-chat Chat Code-generation Text-generation	Getting Started with the Vertex AI Codey APIs - Code Chat This notebook is an introduction to the Vertex AI Codey APIs. It covers the basics of how to use the APIs, including how to create and deploy models, and how to interact with them using the Codey CLI.	Open in Colab Open in GitHub
Language PaLM Python SDK	Getting Started with the PaLM API & Python SDK This notebook provides an introduction to the PaLM API and Python SDK. It covers the basics of how to use the API, including how to create and deploy models, and how to use the API to generate text, translate languages, and write different kinds of creative content.	Open in Colab Open in GitHub
Language prompts	Prompt Design - Best Practices This notebook provides an introduction to prompt design for text-based language models. It covers the basics of prompts, including how they work and how to write them. The notebook also provides tips on how to improve your prompts and avoid common pitfalls.	Open in Colab Open in GitHub
Text-extraction	Text Extraction with Generative Models on Vertex AI This notebook demonstrates how to use generative models to extract text from images. It uses the text-to-image model from the Vertex AI generative-ai library and the text-extraction model from the Vertex AI text-extraction library.	Open in Colab Open in GitHub
Text-classification	Text Classification with Generative Models on Vertex AI This notebook demonstrates how to use generative models to perform text classification on Vertex AI. It covers the following topics: * Preparing data * Training a model * Deploying a model * Using a model to classify text	Open in Colab Open in GitHub
Chain of thought React	Chain of Thought & ReAct This notebook introduces Chain of Thought and ReAct, two tools that can be used to improve the performance of reinforcement learning algorithms. Chain of Thought is a technique that can be used to improve the efficiency of value iteration, while ReAct is a technique that can be used to improve the stability of actor-critic algorithms.	Open in Colab Open in GitHub
Language prompts Ideation	Ideation with Generative Models on Vertex AI This notebook demonstrates how to use generative models to generate text, images, and code. It also shows how to use Vertex AI to deploy and manage generative models.	Open in Colab Open in GitHub
Summarization	Text Summarization with Generative Models on Vertex AI This notebook demonstrates how to use Vertex AI to train and deploy a text summarization model. It uses the BART model, which is a large language model that has been pre-trained on a massive dataset of text. The model is then fine-tuned on a dataset of text summaries, and can then be used to generate summaries of new text.	Open in Colab Open in GitHub
Question-answering	Question Answering with Generative Models on Vertex AI This notebook demonstrates how to use generative models to answer open-domain questions. It uses the Vertex AI Transformer model to generate text based on a given question.	Open in Colab Open in GitHub
Text-generation Foundation-model Tuning Deploy	Tuning and deploy a foundation model This notebook shows how to tune a foundation model using Vertex AI. It also shows how to deploy the tuned model to a Vertex AI endpoint.	Open in Colab Open in GitHub
Document-summarization Summarization	Text Summarization of Large Documents This notebook demonstrates how to use the t5 large model to summarize large documents. The model is trained on a massive dataset of text and code, and it can generate summaries that are both accurate and concise.	Open in Colab Open in GitHub
Document-summarization LangChain Summarization	Text Summarization of Large Documents using LangChain 🦜🔗 This notebook demonstrates how to use the LangChain model to summarize large documents. LangChain is a large language model that can generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way.	Open in Colab Open in GitHub
Document-summarization Document AI Language-model Summarization Text-summarization	Summarization with Large Documents using Document AI and PaLM APIs This notebook demonstrates how to use Document AI and PaLM APIs to summarize large documents. It also shows how to use the Document AI API to extract entities and key phrases from a document.	Open in Colab Open in GitHub
Chatbot Text-generation	GroceryBot, a sample grocery and recipe assistant - RAG + ReAct This notebook is about a sample grocery and recipe assistant that uses RAG and ReAct. It can help you find recipes, create shopping lists, and answer questions about food.	Open in Colab Open in GitHub
Question-answering Document-QA LangChain	Question Answering with Large Documents using LangChain 🦜🔗 This notebook demonstrates how to use the LangChain model to build a question answering system that can answer questions about long documents. The model is trained on a large corpus of text and can be used to answer questions about any topic.	Open in Colab Open in GitHub
Question-answering Document-QA LangChain Vector Search	Question Answering with Documents using LangChain 🦜️🔗 and Vertex AI Vector Search This notebook demonstrates how to use LangChain and Vertex AI Vector Search (previously Matching Engine) to build a question answering system for documents. The system can answer questions about entities, dates, and numbers in documents.	Open in Colab Open in GitHub
Document AI Question-answering PaLM	Question answering with Documents using Document AI, Pandas, and PaLM This notebook demonstrates how to use Document AI, Pandas, and PaLM to build a question answering system. It first uses Document AI to extract structured data from a document, then uses Pandas to create a dataframe from the extracted data, and finally uses PaLM to generate answers to questions about the data.	Open in Colab Open in GitHub
Question-answering Document-QA	Question Answering with Large Documents This notebook demonstrates how to use the Vertex AI Question Answering service to build a question answering model that can answer questions from large documents. The model is trained on a dataset of Wikipedia articles and can answer questions about a variety of topics.	Open in Colab Open in GitHub
Image Generation	Product Description Generator From Image This notebook demonstrates how to generate product descriptions from images using a text-to-image model. The model is trained on a dataset of product images and their corresponding descriptions.	Open in Colab Open in GitHub
Generation Retail LangChain	DescriptionGen: SEO-optimized product description generation for retail using LangChain 🦜🔗 This notebook demonstrates how to use the LangChain model to generate SEO-optimized product descriptions for retail. The model takes as input a list of product attributes and outputs a short description that highlights the key features of the product.	Open in Colab Open in GitHub
BigQuery DataFrames Text-generation	BigQuery DataFrames ML: Drug Name Generation This notebook demonstrates how to use BigQuery DataFrames ML to generate drug names. It uses a pre-trained language model to generate text, and then filters the results to remove drug names that are already in use.	Open in Colab Open in GitHub
BigQuery DataFrames Code-generation	Use BigQuery DataFrames with Generative AI for code generation This notebook demonstrates how to use BigQuery DataFrames with Generative AI for code generation. It shows how to use a pre-trained language model to generate code that transforms a BigQuery table into a Pandas DataFrame.	Open in Colab Open in GitHub
BigQuery Language-model	Using Vertex AI LLMs with data in BigQuery This notebook demonstrates how to use Vertex AI LLMs with data in BigQuery. It shows how to load data from BigQuery, create an LLM model, and then use the model to generate text based on the data.	Open in Colab Open in GitHub
Embeddings Similarity Visualization	Visualizing embedding similarity from text documents using t-SNE plots This notebook demonstrates how to visualize embedding similarity from text documents using t-SNE plots. It uses a dataset of movie reviews from the [IMDB dataset](https://datasets.imdbws.com/).	Open in Colab Open in GitHub
Text-embeddings Vector Search	Getting Started with Text Embeddings + Vertex AI Vector Search This notebook provides an introduction to text embeddings and how to use them with Vertex AI Vector Search. It covers the basics of text embeddings, how to train them, and how to use them to perform vector search.	Open in Colab Open in GitHub
Embeddings Vector Search	Vertex AI Vector Search Quickstart This notebook is a quickstart for using Vertex AI Vector Search. It covers the basics of vector search, including how to create a vector index, how to upload data to the index, and how to perform vector search queries.	Open in Colab Open in GitHub
Imagen 2 image generation	Image Generation with Imagen on Vertex AI In this notebook, you explore the image generation features of Imagen using the Vertex AI SDK for Python. Learn more about Imagen's image generation feature.	Open in Colab Open in GitHub
Imagen 2 image generation Gemini 1.0 Pro text generation Gemini 1.0 Pro output text formatting	Create high quality visual assets with Imagen and Gemini 1.0 Pro In this notebook, you create high quality visual assets for a restaurant menu using Imagen and Gemini 1.0 Pro. Learn more about image generation and multimodal models.	Open in Colab Open in GitHub
Imagen 2 image editing	Create high quality visual assets with Imagen 2 edit using automatically generated mask areas In this notebook, you will be exploring the image editing features of Imagen using the Vertex AI SDK for Python.	Open in Colab Open in GitHub
Imagen image Visual Question Answering (VQA)	Visual Question Answering (VQA) with Imagen on Vertex AI This notebook demonstrates how to use Imagen to generate images that answer given questions. It also shows how to deploy a model to Vertex AI and use it to generate images in response to user-provided questions.	Open in Colab Open in GitHub
Imagen image captioning	Visual captioning with Imagen on Vertex AI This notebook demonstrates how to use Imagen, a large language model for image generation, to generate captions for images. It also shows how to deploy the model on Vertex AI.	Open in Colab Open in GitHub

What's next

• Learn about LLMs, Vertex AI, and PaLM models with notebook tutorials
• Explore more resources in the Generative AI GitHub repo
• See other Vertex AI notebook tutorials in the Tutorials overview