Try the Vertex AI Gemini API

This page shows you how to quickly start sending requests to the Vertex AI Gemini API by using the Google Cloud console, a programming language SDK, or the REST API.

New to Google Cloud

Get set up on Google Cloud

If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers get $300 in free credits to run, test, and deploy workloads. The setup process is only three short steps:

Use the following button to create an account. After you're done, return to this page to complete this beginner tutorial. To use all features available on this site, sign in using your account.

Create an account

For more information on getting set up on Google Cloud, see Get set up on Google Cloud.

Send a request to the Vertex AI Gemini API

To see the instructions for sending a request to the Vertex AI Gemini API, select one of the following tabs:

Python

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

2. In Cloud Shell, install or update the Vertex AI SDK for Python by running the following command:

   pip install "google-cloud-aiplatform>=1.38"
   

Send a prompt request. Replace PROJECT_ID with the ID of your Google Cloud project.

# TODO(developer): Vertex AI SDK - uncomment below & run
# pip3 install --upgrade --user google-cloud-aiplatform
# gcloud auth application-default login

import vertexai
from vertexai.generative_models import GenerativeModel, Part

# Initialize Vertex AI
vertexai.init(project=project_id, location=location)
# Load the model
multimodal_model = GenerativeModel(model_name="gemini-1.0-pro-vision-001")
# Query the model
response = multimodal_model.generate_content(
    [
        # Add an example image
        Part.from_uri(
            "gs://generativeai-downloads/images/scones.jpg", mime_type="image/jpeg"
        ),
        # Add an example query
        "what is shown in this image?",
    ]
)
print(response)
return response.text

To learn how to install or update the Vertex AI SDK for Python, see Install the Vertex AI SDK for Python. For more information, see the Vertex AI SDK for Python API reference documentation.

Node.js

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

2. In Cloud Shell, install or update the Vertex AI SDK for Node.js by running the following command:

   npm install @google-cloud/vertexai
   

Send a prompt request. Replace PROJECT_ID with the ID of your Google Cloud project.

const {VertexAI} = require('@google-cloud/vertexai');

/**
 * TODO(developer): Update these variables before running the sample.
 */
async function createNonStreamingMultipartContent(
  projectId = 'PROJECT_ID',
  location = 'us-central1',
  model = 'gemini-1.0-pro-vision',
  image = 'gs://generativeai-downloads/images/scones.jpg',
  mimeType = 'image/jpeg'
) {
  // Initialize Vertex with your Cloud project and location
  const vertexAI = new VertexAI({project: projectId, location: location});

  // Instantiate the model
  const generativeVisionModel = vertexAI.getGenerativeModel({
    model: model,
  });

  // For images, the SDK supports both Google Cloud Storage URI and base64 strings
  const filePart = {
    fileData: {
      fileUri: image,
      mimeType: mimeType,
    },
  };

  const textPart = {
    text: 'what is shown in this image?',
  };

  const request = {
    contents: [{role: 'user', parts: [filePart, textPart]}],
  };

  console.log('Prompt Text:');
  console.log(request.contents[0].parts[1].text);

  console.log('Non-Streaming Response Text:');
  // Create the response stream
  const responseStream =
    await generativeVisionModel.generateContentStream(request);

  // Wait for the response stream to complete
  const aggregatedResponse = await responseStream.response;

  // Select the text from the response
  const fullTextResponse =
    aggregatedResponse.candidates[0].content.parts[0].text;

  console.log(fullTextResponse);
}

For more information on installing and using the Vertex AI Node.js SDK, see the Vertex AI SDK for Node.js reference documentation.

Java

1. Set up your Java Development Environment.

2. Authenticate by running the following command. Replace PROJECT_ID with your Google Cloud project ID and ACCOUNT with your Google Cloud username.

   gcloud config set project PROJECT_ID &&
   gcloud auth login ACCOUNT
   

3. Add google-cloud-vertexai as your dependency:

   <!--If you are using Maven with BOM, add the following in your pom.xml-->
   <dependencyManagement>
     <dependencies>
       <dependency>
         <groupId>com.google.cloud</groupId>
         <artifactId>libraries-bom</artifactId>
         <version>26.32.0</version>
         <type>pom</type>
         <scope>import</scope>
       </dependency>
     </dependencies>
   </dependencyManagement>
   <dependencies>
     <dependency>
       <groupId>com.google.cloud</groupId>
       <artifactId>google-cloud-vertexai</artifactId>
     </dependency>
   </dependencies>
   
   <!--If you are using Maven without BOM, add the following to your pom.xml-->
   <dependency>
     <groupId>com.google.cloud</groupId>
     <artifactId>google-cloud-vertexai</artifactId>
     <version>0.4.0</version>
   </dependency>
   
   <!--If you are using Gradle without BOM, add the following to your build.gradle-->
   implementation 'com.google.cloud:google-cloud-vertexai:0.4.0'
   

Send a prompt request. Set projectID to your Google Cloud project ID.

import com.google.cloud.vertexai.VertexAI;
import com.google.cloud.vertexai.api.GenerateContentResponse;
import com.google.cloud.vertexai.generativeai.ContentMaker;
import com.google.cloud.vertexai.generativeai.GenerativeModel;
import com.google.cloud.vertexai.generativeai.PartMaker;
import java.io.IOException;

public class Quickstart {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-google-cloud-project-id";
    String location = "us-central1";
    String modelName = "gemini-1.0-pro-vision";

    String output = quickstart(projectId, location, modelName);
    System.out.println(output);
  }

  // Analyzes the provided Multimodal input.
  public static String quickstart(String projectId, String location, String modelName)
      throws IOException {
    // Initialize client that will be used to send requests. This client only needs
    // to be created once, and can be reused for multiple requests.
    try (VertexAI vertexAI = new VertexAI(projectId, location)) {
      String imageUri = "gs://generativeai-downloads/images/scones.jpg";

      GenerativeModel model = new GenerativeModel(modelName, vertexAI);
      GenerateContentResponse response = model.generateContent(ContentMaker.fromMultiModalData(
          PartMaker.fromMimeTypeAndData("image/jpg", imageUri),
          "What's in this photo"
      ));

      return response.toString();
    }
  }
}

For more information on installing and using the Vertex AI Java Development Kit (JDK), see the Vertex AI JDK reference documentation.

Go

1. Prepare your environment for Go development.

2. Review the available Vertex AI API Go packages to determine which package best meets your project's needs:

   • Package cloud.google.com/go/vertexai (recommended)

     vertexai is a human authored package that provides access to common capabilities and features.

     This package is recommended as the starting point for most developers building with the Vertex AI API. To access capabilities and features not yet covered by this package, use the auto-generated aiplatform instead.

   • Package cloud.google.com/go/aiplatform

     aiplatform is an auto-generated package.

     This package is intended for projects that require access to Vertex AI API capabilities and features not yet provided by the human authored vertexai package.

3. Install the desired Go package based on your project's needs by running one of the following commands:

   # Human authored package. Recommended for most developers.
   go get cloud.google.com/go/vertexai
   
   
   # Auto-generated package.
   go get cloud.google.com/go/aiplatform
   

Send a prompt request. Replace PROJECT_ID with the ID of your Google Cloud project.

import (
	"context"
	"encoding/json"
	"fmt"
	"io"

	"cloud.google.com/go/vertexai/genai"
)

var projectId = "PROJECT_ID"
var region = "us-central1"

func tryGemini(w io.Writer, projectId string, region string, modelName string) error {

	ctx := context.Background()
	client, err := genai.NewClient(ctx, projectId, region)
	gemini := client.GenerativeModel(modelName)

	img := genai.FileData{
		MIMEType: "image/jpeg",
		FileURI:  "gs://generativeai-downloads/images/scones.jpg",
	}
	prompt := genai.Text("What is in this image?")
	resp, err := gemini.GenerateContent(ctx, img, prompt)
	if err != nil {
		return fmt.Errorf("error generating content: %w", err)
	}
	rb, _ := json.MarshalIndent(resp, "", "  ")
	fmt.Fprintln(w, string(rb))
	return nil
}

For more information on installing and using the Vertex AI SDK for Go, see the Vertex AI SDK for Go reference documentation.

C#

Before trying this sample, follow the C# setup instructions in the Vertex AI quickstart using client libraries. For more information, see the Vertex AI C# API reference documentation.

To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

using Google.Api.Gax.Grpc;
using Google.Cloud.AIPlatform.V1;
using System.Collections.Generic;
using System.Text;
using System.Threading.Tasks;

public class GeminiQuickstart
{
    public async Task<string> GenerateContent(
        string projectId = "your-project-id",
        string location = "us-central1",
        string publisher = "google",
        string model = "gemini-1.0-pro-vision"
    )
    {
        // Create client
        var predictionServiceClient = new PredictionServiceClientBuilder
        {
            Endpoint = $"{location}-aiplatform.googleapis.com"
        }.Build();

        // Prompt
        string prompt = "What's in this photo";
        string imageUri = "gs://generativeai-downloads/images/scones.jpg";

        // Initialize request argument(s)
        var content = new Content
        {
            Role = "USER"
        };
        content.Parts.AddRange(new List<Part>()
        {
            new() {
                Text = prompt
            },
            new() {
                FileData = new() {
                    MimeType = "image/png",
                    FileUri = imageUri
                }
            }
        });

        var generateContentRequest = new GenerateContentRequest
        {
            Model = $"projects/{projectId}/locations/{location}/publishers/{publisher}/models/{model}",
            GenerationConfig = new GenerationConfig
            {
                Temperature = 0.4f,
                TopP = 1,
                TopK = 32,
                MaxOutputTokens = 2048
            }
        };
        generateContentRequest.Contents.Add(content);

        // Make the request, returning a streaming response
        using PredictionServiceClient.StreamGenerateContentStream response = predictionServiceClient.StreamGenerateContent(generateContentRequest);

        StringBuilder fullText = new();

        // Read streaming responses from server until complete
        AsyncResponseStream<GenerateContentResponse> responseStream = response.GetResponseStream();
        await foreach (GenerateContentResponse responseItem in responseStream)
        {
            fullText.Append(responseItem.Candidates[0].Content.Parts[0].Text);
        }

        return fullText.ToString();
    }
}

REST

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

2. Configure environment variables by entering the following. Replace PROJECT_ID with the ID of your Google Cloud project.

   MODEL_ID="gemini-1.0-pro-vision"
   PROJECT_ID="PROJECT_ID"
   

3. Provision the endpoint:

   gcloud beta services identity create --service=aiplatform.googleapis.com --project=PROJECT_ID
   

4. Send a prompt request by entering the following curl command:

   curl \
   -X POST \
   -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
   -H "Content-Type: application/json" \
   https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:streamGenerateContent -d \
   $'{
     "contents": {
       "role": "user",
       "parts": [
         {
         "fileData": {
           "mimeType": "image/jpeg",
           "fileUri": "gs://generativeai-downloads/images/scones.jpg"
           }
         },
         {
           "text": "Describe this picture."
         }
       ]
     }
   }'
   

5. If asked to authorize Cloud Shell, click Authorize.

   The model returns a response. Note that the response is generated in sections with each section separately evaluated for safety.

Console

Use Vertex AI Studio to quickly design and iterate on your prompts. After your prompt is ready, you can get the code for your prompt in any of the supported programming languages.

1. In the Google Cloud console, go to the Vertex AI Studio page.

   Vertex AI Studio

2. Click Multimodal.

3. Under Sample prompts, locate the prompt titled Extract text from images, and click Open.

   The prompt page opens and the prompt is populated in the Prompt field.

4. Submit the prompt by clicking Submit.

   The model returns a response.

5. View the code equivalent of this prompt request by clicking Get code.

What's next

• Learn more about the Vertex AI Gemini API.
• Learn how to design multimodal prompts.
• See the in-depth guides on the Vertex AI Gemini API:
  • Test multimodal prompts.
  • Test chat prompts.
• See the Vertex AI Gemini API reference.
• See the Vertex AI Gemini API Python SDK reference.