Gemini in Vertex AI 的 Model API

借助 Vertex AI API for Gemini，您可以使用 Gemini 模型创建应用。您可以使用该服务创建请求，然后接收响应，以创建适合您的使用场景的应用。以下主题介绍了 Gemini 模型的一些示例使用场景：

• Gemini 1.5 Pro 和 Gemini 1.5 Flash 使用场景
• Gemini 1.0 Pro Vision 使用场景
• Gemini 1.0 Pro 使用场景

创建一个 Google Cloud 账号即可开始使用

如要开始使用 Vertex AI Model API for Gemini，您需要先创建一个 Google Cloud 账号。

创建账号后，您可以使用本文档了解 Gemini 模型的请求正文、模型参数和响应正文，并查看一些示例请求和响应。

准备就绪后，请参阅 Vertex AI API for Gemini 快速入门，了解如何使用编程语言 SDK 或 REST API 向 Vertex AI Gemini API 发送请求。

发送一个 HTTP 请求

下面的各标签页分别展示了如何使用各个版本的 Gemini 模型发送 HTTP 请求：

Gemini 1.5 Pro

POST https://{REGION}-aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/{REGION}/publishers/google/models/gemini-1.5-pro:streamGenerateContent

Gemini 1.5 Flash

POST https://{REGION}-aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/{REGION}/publishers/google/models/gemini-1.5-flash:streamGenerateContent

Gemini 1.0 Pro

POST https://{REGION}-aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/{REGION}/publishers/google/models/gemini-1.0-pro:streamGenerateContent

注意：您可以通过将 ?alt=sse 附加到网址，为 gemini-1.0-pro 启用服务器发送的事件 (SSE)。

Gemini 1.0 Pro Vision

POST https://{REGION}-aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/{REGION}/publishers/google/models/gemini-1.0-pro-vision:streamGenerateContent

如需向模型发送数据流请求，请参阅 streamGenerateContent 方法了解详情。

如需向模型发送非数据流请求，请改用 generateContent 方法。

如需查看支持的区域列表，请参阅可用位置。

模型版本

如需使用自动更新版本，请指定不含尾随版本号的模型名称，例如 gemini-1.0-pro，而不是 gemini-1.0-pro-001。

如需了解详情，请参阅 Gemini 模型版本和生命周期。

请求正文

请求正文中包含结构如下的数据：

{
  "contents": [
    {
      "role": string,
      "parts": [
        {
          // Union field data can be only one of the following:
          "text": string,
          "inlineData": {
            "mimeType": string,
            "data": string
          },
          "fileData": {
            "mimeType": string,
            "fileUri": string
          },
          // End of list of possible types for union field data.

          "videoMetadata": {
            "startOffset": {
              "seconds": integer,
              "nanos": integer
            },
            "endOffset": {
              "seconds": integer,
              "nanos": integer
            }
          }
        }
      ]
    }
  ],
  "systemInstruction": {
    "role": string,
    "parts": [
      {
        "text": string
      }
    ]
  },
  "tools": [
    {
      "functionDeclarations": [
        {
          "name": string,
          "description": string,
          "parameters": {
            object (OpenAPI Object Schema)
          }
        }
      ]
    }
  ],
  "safetySettings": [
    {
      "category": enum (HarmCategory),
      "threshold": enum (HarmBlockThreshold)
    }
  ],
  "generationConfig": {
    "temperature": number,
    "topP": number,
    "topK": number,
    "candidateCount": integer,
    "maxOutputTokens": integer,
    "presencePenalty": float,
    "frequencyPenalty": float,
    "stopSequences": [
      string
    ],
    "responseMimeType": string
  }
}

Gemini 模型参数

您可以在请求正文中使用以下参数：

参数	说明
role	与内容关联的对话中的角色。即使在单轮应用场景中，也需要指定角色。 可接受的值包括： USER：指定由您发送的内容。 MODEL：指定模型的响应。
parts	构成输入的有序部分。部分可能具有不同的 MIME 类型。 如需了解输入限制（例如词元或图片数量上限），请参阅 Google 模型页面上的模型规范部分。 如需计算请求中的词元数量，请参阅获取词元数。
text	要包含在提示中的文本说明或聊天对话框。
inlineData	图片、音频片段或视频片段的序列化字节数据。 对于 gemini-1.0-pro-vision，您可以使用 inlineData 指定最多 1 张图片。如需指定最多 16 张图片，请使用 fileData。
mimeType	在 data 或 fileUri 字段中指定的文件的媒体类型。可接受的值包括： 点击即可展开 MIME 类型 application/pdf audio/mpeg audio/mp3 audio/wav image/png image/jpeg text/plain video/mov video/mpeg video/mp4 video/mpg video/avi video/wmv video/mpegps video/flv 对于 gemini-1.0-pro-vision，视频时长上限为 2 分钟。 对于 Gemini 1.5 Pro，音频文件的时长上限为 8.4 小时，视频文件（不含音频）的时长上限为 1 小时。如需了解详情，请参阅 Gemini 1.5 Pro 媒体要求。 文本文件必须采用 UTF-8 编码。文本文件的内容会计入词元数限制。 图片分辨率没有限制。
data	：要在提示中包含内嵌的图片、PDF 或视频的 base64 编码。添加媒体内嵌时，您还必须指定 MIMETYPE。 大小上限：20MB
fileUri	要包含在提示中的文件的 Cloud Storage URI。存储桶对象必须可公开读取，或者位于发送请求的同一 Google Cloud 项目中。您还必须指定文件的媒体类型 (MIMETYPE)。 对于 gemini-1.5-pro，大小限制为 2GB。 对于 gemini-1.0-pro-vision，大小限制为 20MB。
videoMetadata	可选。对于视频输入，为视频的开始和结束偏移量，采用时长格式。例如，如需指定从 1:00 开始的 10 秒剪辑，请设置 "start_offset": { "seconds": 60 } 和 "end_offset": { "seconds": 70 }。
systemInstruction（预览版）	可选。适用于 gemini-1.5-pro 和 gemini-1.0-pro-002。 有关引导模型获得更好性能的说明。例如，“回答尽可能简明扼要”或“请勿在回答中使用技术词汇”。 text 字符串会计入词元限制。 systemInstruction 的 role 字段会被忽略，不会影响模型的性能。
tools	一段代码，可让系统与外部系统进行交互，以在模型知识和范围之外执行操作或一组操作。
functionDeclarations	一个或多个函数声明。每个函数声明包含一个函数的相关信息，其中包括： name：要调用的函数名称。必须以字母或下划线开头。必须是 a-z、A-Z、0-9 或包含下划线和短划线，长度上限为 64。 description（可选）。函数的说明和用途。模型使用此参数来确定如何以及是否调用函数。为获得最佳结果，我们建议您添加说明。 parameters 此函数的参数，其格式与 OpenAPI 架构兼容。 如需了解详情，请参阅函数调用。
category	要为其配置阈值的安全类别。可接受的值包括： 点击即可展开安全类别 HARM_CATEGORY_SEXUALLY_EXPLICIT HARM_CATEGORY_HATE_SPEECH HARM_CATEGORY_HARASSMENT HARM_CATEGORY_DANGEROUS_CONTENT
threshold	基于概率阻止属于指定安全类别的响应的阈值。 BLOCK_NONE BLOCK_LOW_AND_ABOVE BLOCK_MED_AND_ABOVE BLOCK_ONLY_HIGH
temperature	温度 (temperature) 在生成回复期间用于采样，在应用 topP 和 topK 时会生成回复。温度可以控制词元选择的随机性。 较低的温度有利于需要更少开放性或创造性回复的提示，而较高的温度可以带来更具多样性或创造性的结果。温度为 0 表示始终选择概率最高的词元。在这种情况下，给定提示的回复大多是确定的，但可能仍然有少量变化。 如果模型返回的回答过于笼统、过于简短，或者模型给出后备回答，请尝试提高温度。 gemini-1.5-pro 的温度范围：0.0 - 2.0（默认值：1.0） gemini-1.0-pro-vision 的温度范围：0.0 - 1.0（默认值：0.4） gemini-1.0-pro-002 的温度范围：0.0 - 2.0（默认值：1.0） gemini-1.0-pro-001 的温度范围：0.0 - 1.0（默认值：0.9）
maxOutputTokens	回复中可生成的词元数量上限。词元约为 4 个字符。100 个词元对应大约 60-80 个单词。 指定较低的值可获得较短的回答，指定较高的值可获得可能较长的回答。 如需了解每个版本的模型的输出词元数上限，请参阅 Google 模型页面上的模型规范部分。默认情况下，Google 会将模型的输出词元数上限用作可生成的词元数上限。
topK	Top-K 可更改模型选择输出词元的方式。如果 top-K 设为 1，表示下一个所选词元是模型词汇表的所有词元中概率最高的词元（也称为贪心解码）。如果 top-K 设为 3，则表示系统将从 3 个概率最高的词元（通过温度确定）中选择下一个词元。 在每个词元选择步中，系统都会对概率最高的 top-K 词元进行采样。然后，系统会根据 top-P 进一步过滤词元，并使用温度采样选择最终的词元。 指定较低的值可获得随机程度较低的回答，指定较高的值可获得随机程度较高的回答。 范围：1-40 gemini-1.0-pro 和 gemini-1.5-pro 不支持 topK gemini-1.0-pro-vision 的默认值： 32
topP	Top-P 可更改模型选择输出词元的方式。系统会按照概率从最高（见 top-K）到最低的顺序选择词元，直到所选词元的概率总和等于 top-P 的值。例如，如果词元 A、B 和 C 的概率分别为 0.3、0.2 和 0.1，并且 top-P 的值为 0.5，则模型将选择 A 或 B 作为下一个词元（通过温度确定），并会排除 C 作为候选词元。 指定较低的值可获得随机程度较低的回答，指定较高的值可获得随机程度较高的回答。 范围：0.0 - 1.0 gemini-1.5-pro：0.94 gemini-1.0-pro 的默认值：1 gemini-1.0-pro-vision的默认值：1
frequencyPenalty	正值会惩罚生成的文本中反复出现的词元，从而降低重复内容概率。 frequencyPenalty 的最大值为 2.0，但不包括该数值；最小值为 -2.0。
presencePenalty	正值会惩罚生成的文本中已存在的词元，从而增加生成更多样化内容的概率。 presencePenalty 的最大值为 2.0，但不包括该数值；最小值为 -2.0。
candidateCount	要返回的响应变体数量。 该值必须为 1。
stopSequences	指定一个字符串列表，告知模型在响应中遇到其中一个字符串时，停止生成文本。如果某个字符串在响应中多次出现，则响应会在首次出现的位置截断。字符串区分大小写。 例如，未指定 stopSequences 时，如果下面的内容是返回的回答： public static string reverse(string myString) 则返回的回答为以下内容，其中 stopSequences 设置为 ["Str", "reverse"]： public static string 列表中最多 5 项。
responseMimeType（预览版）	可选。适用于 gemini-1.5-pro。 生成的候选文本的输出格式。 支持的 MIME 类型： text/plain：（默认）文本输出。 application/json：候选项中的 JSON 响应。

响应正文

{
  "candidates": [
    {
      "content": {
        "parts": [
          {
            "text": string
          }
        ]
      },
      "finishReason": enum (FinishReason),
      "safetyRatings": [
        {
          "category": enum (HarmCategory),
          "probability": enum (HarmProbability),
          "blocked": boolean
        }
      ],
      "citationMetadata": {
        "citations": [
          {
            "startIndex": integer,
            "endIndex": integer,
            "uri": string,
            "title": string,
            "license": string,
            "publicationDate": {
              "year": integer,
              "month": integer,
              "day": integer
            }
          }
        ]
      }
    }
  ],
  "usageMetadata": {
    "promptTokenCount": integer,
    "candidatesTokenCount": integer,
    "totalTokenCount": integer
  }
}

响应元素	说明
text	生成的文本。
finishReason	模型停止生成词元的原因。如果为空，则模型尚未停止生成词元。由于回答使用上下文提示，因此无法更改模型停止生成词元的行为。 FINISH_REASON_UNSPECIFIED 未指定完成原因。 FINISH_REASON_STOP 模型的自然停止点或提供的停止序列。 FINISH_REASON_MAX_TOKENS 已达到请求中指定的词元数量上限。 FINISH_REASON_SAFETY 由于出于安全原因标记了答复，词元生成已停止。请注意，如果内容过滤器阻止输出，则 Candidate.content 为空。 FINISH_REASON_RECITATION 由于响应因未经授权的引用而进行标记，因此词元生成操作已停止。 FINISH_REASON_OTHER 停止词元的所有其他原因
category	要为其配置阈值的安全类别。可接受的值包括： 点击即可展开安全类别 HARM_CATEGORY_SEXUALLY_EXPLICIT HARM_CATEGORY_HATE_SPEECH HARM_CATEGORY_HARASSMENT HARM_CATEGORY_DANGEROUS_CONTENT
probability	内容中的有害概率级别。 HARM_PROBABILITY_UNSPECIFIED NEGLIGIBLE LOW MEDIUM HIGH
blocked	一个与安全属性关联的布尔值标志，用于指示模型的输入或输出是否被阻止。
startIndex	一个整数，用于指定引用在内容中的起始位置。
endIndex	一个整数，用于指定引用在 content 中的结束位置。
url	引用来源的网址。网址来源的示例可能是新闻网站或 GitHub 代码库。
title	引用来源的标题。来源标题的示例可能是新闻报道或书籍标题。
license	与引用关联的许可。
publicationDate	引用的发布日期。其有效格式为 YYYY、YYYY-MM、YYYY-MM-DD。
promptTokenCount	请求中的词元数量。
candidatesTokenCount	响应中的词元数量。
totalTokenCount	请求和响应中的词元数量。

采样请求

文本

REST

如需使用启用了服务器发送事件 (SSE) 的 Vertex AI API 测试文本提示，请向发布方模型端点发送 POST 请求，并在网址末尾添加 ?alt=sse。

在使用任何请求数据之前，请先进行以下替换：

• PROJECT_ID：您的项目 ID。

如需了解其他字段，请查看请求正文表。

HTTP 方法和网址：

POST https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-1.0-pro:streamGenerateContent?alt=sse

请求 JSON 正文：

{
  "contents": {
    "role": "user",
    "parts": {
        "text": "Give me a recipe for banana bread."
    }
  },
  "safety_settings": {
    "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
    "threshold": "BLOCK_LOW_AND_ABOVE"
  },
  "generation_config": {
    "temperature": 0.2,
    "topP": 0.8,
    "topK": 40
  }
}

如需发送请求，请选择以下方式之一：

curl

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI，或者使用了 Cloud Shell，这会使您自动登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

将请求正文保存在名为 request.json 的文件中，然后执行以下命令：

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-1.0-pro:streamGenerateContent?alt=sse"

PowerShell

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

将请求正文保存在名为 request.json 的文件中，然后执行以下命令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-1.0-pro:streamGenerateContent?alt=sse" | Select-Object -Expand Content

您应该会收到类似示例响应的 JSON 响应。

聊天

另请参阅发送聊天提示请求 (Gemini)。

REST

如需使用启用了服务器发送事件 (SSE) 的 Vertex AI API 测试聊天提示，请向发布方模型端点发送 POST 请求，并在网址末尾添加 ?alt=sse。

在使用任何请求数据之前，请先进行以下替换：

• PROJECT_ID：您的项目 ID。

如需了解其他字段，请查看请求正文表。

HTTP 方法和网址：

POST https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-1.0-pro:streamGenerateContent?alt=sse

请求 JSON 正文：

{
  "contents": [
    {
      "role": "USER",
      "parts": { "text": "Hello!" }
    },
    {
      "role": "MODEL",
      "parts": { "text": "Argh! What brings ye to my ship?" }
    },
    {
      "role": "USER",
      "parts": { "text": "Wow! You are a real-life priate!" }
    }
  ],
  "safety_settings": {
    "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
    "threshold": "BLOCK_LOW_AND_ABOVE"
  },
  "generation_config": {
    "temperature": 0.2,
    "topP": 0.8,
    "topK": 40,
    "maxOutputTokens": 200,
  }
}

如需发送请求，请选择以下方式之一：

curl

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI，或者使用了 Cloud Shell，这会使您自动登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

将请求正文保存在名为 request.json 的文件中，然后执行以下命令：

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-1.0-pro:streamGenerateContent?alt=sse"

PowerShell

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

将请求正文保存在名为 request.json 的文件中，然后执行以下命令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-1.0-pro:streamGenerateContent?alt=sse" | Select-Object -Expand Content

您应该会收到类似示例响应的 JSON 响应。

Python

如需了解如何安装或更新 Python 版 Vertex AI SDK，请参阅安装 Python 版 Vertex AI SDK。 如需了解详情，请参阅 Python API 参考文档。

import vertexai

from vertexai.generative_models import GenerativeModel, ChatSession

# TODO(developer): Update and un-comment below line
# project_id = "PROJECT_ID"

vertexai.init(project=project_id, location="us-central1")

model = GenerativeModel(model_name="gemini-1.0-pro-002")

chat = model.start_chat()

def get_chat_response(chat: ChatSession, prompt: str) -> str:
    text_response = []
    responses = chat.send_message(prompt, stream=True)
    for chunk in responses:
        text_response.append(chunk.text)
    return "".join(text_response)

prompt = "Hello."
print(get_chat_response(chat, prompt))

prompt = "What are all the colors in a rainbow?"
print(get_chat_response(chat, prompt))

prompt = "Why does it appear when it rains?"
print(get_chat_response(chat, prompt))

Node.js

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Node.js 设置说明执行操作。 如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。 如需了解详情，请参阅为本地开发环境设置身份验证。

const {VertexAI} = require('@google-cloud/vertexai');

/**
 * TODO(developer): Update these variables before running the sample.
 */
async function createStreamChat(
  projectId = 'PROJECT_ID',
  location = 'us-central1',
  model = 'gemini-1.5-flash-001'
) {
  // Initialize Vertex with your Cloud project and location
  const vertexAI = new VertexAI({project: projectId, location: location});

  // Instantiate the model
  const generativeModel = vertexAI.getGenerativeModel({
    model: model,
  });

  const chat = generativeModel.startChat({});
  const chatInput1 = 'How can I learn more about that?';

  console.log(`User: ${chatInput1}`);

  const result1 = await chat.sendMessageStream(chatInput1);
  for await (const item of result1.stream) {
    console.log(item.candidates[0].content.parts[0].text);
  }
}

Java

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Java 设置说明执行操作。 如需了解详情，请参阅 Vertex AI Java API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。 如需了解详情，请参阅为本地开发环境设置身份验证。

import com.google.cloud.vertexai.VertexAI;
import com.google.cloud.vertexai.api.GenerateContentResponse;
import com.google.cloud.vertexai.generativeai.ChatSession;
import com.google.cloud.vertexai.generativeai.GenerativeModel;
import com.google.cloud.vertexai.generativeai.ResponseHandler;
import java.io.IOException;

public class ChatDiscussion {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-google-cloud-project-id";
    String location = "us-central1";
    String modelName = "gemini-1.0-pro-002";

    chatDiscussion(projectId, location, modelName);
  }

  // Ask interrelated questions in a row using a ChatSession object.
  public static void chatDiscussion(String projectId, String location, String modelName)
      throws IOException {
    // Initialize client that will be used to send requests. This client only needs
    // to be created once, and can be reused for multiple requests.
    try (VertexAI vertexAI = new VertexAI(projectId, location)) {
      GenerateContentResponse response;

      GenerativeModel model = new GenerativeModel(modelName, vertexAI);
      // Create a chat session to be used for interactive conversation.
      ChatSession chatSession = new ChatSession(model);

      response = chatSession.sendMessage("Hello.");
      System.out.println(ResponseHandler.getText(response));

      response = chatSession.sendMessage("What are all the colors in a rainbow?");
      System.out.println(ResponseHandler.getText(response));

      response = chatSession.sendMessage("Why does it appear when it rains?");
      System.out.println(ResponseHandler.getText(response));
      System.out.println("Chat Ended.");
    }
  }
}

Go

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Go 设置说明执行操作。 如需了解详情，请参阅 Vertex AI Go API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。 如需了解详情，请参阅为本地开发环境设置身份验证。

import (
	"context"
	"encoding/json"
	"fmt"
	"io"

	"cloud.google.com/go/vertexai/genai"
)

func makeChatRequests(ctx context.Context, w io.Writer, projectID, region, modelName string) error {
	client, err := genai.NewClient(ctx, projectID, region)

	if err != nil {
		return fmt.Errorf("error creating client: %w", err)
	}
	defer client.Close()

	gemini := client.GenerativeModel(modelName)
	chat := gemini.StartChat()

	send := func(message string) error {
		r, err := chat.SendMessage(ctx, genai.Text(message))
		if err != nil {
			return err
		}
		rb, err := json.MarshalIndent(r, "", "  ")
		if err != nil {
			return err
		}
		fmt.Fprintln(w, string(rb))
		return nil
	}

	if err := send("Hello"); err != nil {
		return err
	}
	if err := send("What are all the colors in a rainbow?"); err != nil {
		return err
	}
	return send("Why does it appear when it rains?")
}

多模态

另请参阅发送多模态提示请求。

REST

如需使用 Vertex AI API 测试多模态提示，请向发布者模型端点发送 POST 请求。

在使用任何请求数据之前，请先进行以下替换：

• PROJECT_ID：您的项目 ID。

如需了解其他字段，请查看请求正文表。

HTTP 方法和网址：

POST https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-1.0-pro-vision:streamGenerateContent

请求 JSON 正文：

{
  "contents": {
    "role": "user",
    "parts": [
      {
        "fileData": {
          "mimeType": "image/jpeg",
          "fileUri": "gs://cloud-samples-data/ai-platform/flowers/daisy/10559679065_50d2b16f6d.jpg"
        }
      },
      {
        "text": "Describe this picture."
      }
    ]
  },
  "safety_settings": {
    "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
    "threshold": "BLOCK_LOW_AND_ABOVE"
  },
  "generation_config": {
    "temperature": 0.4,
    "topP": 1.0,
    "topK": 32,
    "maxOutputTokens": 2048
  }
}

如需发送请求，请选择以下方式之一：

curl

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI，或者使用了 Cloud Shell，这会使您自动登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

将请求正文保存在名为 request.json 的文件中，然后执行以下命令：

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-1.0-pro-vision:streamGenerateContent"

PowerShell

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

将请求正文保存在名为 request.json 的文件中，然后执行以下命令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-1.0-pro-vision:streamGenerateContent" | Select-Object -Expand Content

您应该会收到类似示例响应的 JSON 响应。

Python

如需了解如何安装或更新 Python 版 Vertex AI SDK，请参阅安装 Python 版 Vertex AI SDK。 如需了解详情，请参阅 Python API 参考文档。

import vertexai

from vertexai.generative_models import GenerativeModel, Part

# TODO(developer): Update and un-comment below line
# project_id = "PROJECT_ID"

vertexai.init(project=project_id, location="us-central1")

# Load images from Cloud Storage URI
image_file1 = Part.from_uri(
    "gs://cloud-samples-data/vertex-ai/llm/prompts/landmark1.png",
    mime_type="image/png",
)
image_file2 = Part.from_uri(
    "gs://cloud-samples-data/vertex-ai/llm/prompts/landmark2.png",
    mime_type="image/png",
)
image_file3 = Part.from_uri(
    "gs://cloud-samples-data/vertex-ai/llm/prompts/landmark3.png",
    mime_type="image/png",
)

model = GenerativeModel(model_name="gemini-1.5-flash-001")
response = model.generate_content(
    [
        image_file1,
        "city: Rome, Landmark: the Colosseum",
        image_file2,
        "city: Beijing, Landmark: Forbidden City",
        image_file3,
    ]
)
print(response.text)

Node.js

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Node.js 设置说明执行操作。 如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。 如需了解详情，请参阅为本地开发环境设置身份验证。

const {VertexAI} = require('@google-cloud/vertexai');
const axios = require('axios');

async function getBase64(url) {
  const image = await axios.get(url, {responseType: 'arraybuffer'});
  return Buffer.from(image.data).toString('base64');
}

/**
 * TODO(developer): Update these variables before running the sample.
 */
async function sendMultiModalPromptWithImage(
  projectId = 'PROJECT_ID',
  location = 'us-central1',
  model = 'gemini-1.0-pro-vision-001'
) {
  // For images, the SDK supports base64 strings
  const landmarkImage1 = await getBase64(
    'https://storage.googleapis.com/cloud-samples-data/vertex-ai/llm/prompts/landmark1.png'
  );
  const landmarkImage2 = await getBase64(
    'https://storage.googleapis.com/cloud-samples-data/vertex-ai/llm/prompts/landmark2.png'
  );
  const landmarkImage3 = await getBase64(
    'https://storage.googleapis.com/cloud-samples-data/vertex-ai/llm/prompts/landmark3.png'
  );

  // Initialize Vertex with your Cloud project and location
  const vertexAI = new VertexAI({project: projectId, location: location});

  const generativeVisionModel = vertexAI.getGenerativeModel({
    model: model,
  });

  // Pass multimodal prompt
  const request = {
    contents: [
      {
        role: 'user',
        parts: [
          {
            inlineData: {
              data: landmarkImage1,
              mimeType: 'image/png',
            },
          },
          {
            text: 'city: Rome, Landmark: the Colosseum',
          },

          {
            inlineData: {
              data: landmarkImage2,
              mimeType: 'image/png',
            },
          },
          {
            text: 'city: Beijing, Landmark: Forbidden City',
          },
          {
            inlineData: {
              data: landmarkImage3,
              mimeType: 'image/png',
            },
          },
        ],
      },
    ],
  };

  // Create the response
  const response = await generativeVisionModel.generateContent(request);
  // Wait for the response to complete
  const aggregatedResponse = await response.response;
  // Select the text from the response
  const fullTextResponse =
    aggregatedResponse.candidates[0].content.parts[0].text;

  console.log(fullTextResponse);
}

Java

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Java 设置说明执行操作。 如需了解详情，请参阅 Vertex AI Java API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。 如需了解详情，请参阅为本地开发环境设置身份验证。

import com.google.cloud.vertexai.VertexAI;
import com.google.cloud.vertexai.api.Content;
import com.google.cloud.vertexai.api.GenerateContentResponse;
import com.google.cloud.vertexai.generativeai.ContentMaker;
import com.google.cloud.vertexai.generativeai.GenerativeModel;
import com.google.cloud.vertexai.generativeai.PartMaker;
import com.google.cloud.vertexai.generativeai.ResponseHandler;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;

public class MultimodalMultiImage {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-google-cloud-project-id";
    String location = "us-central1";
    String modelName = "gemini-1.0-pro-vision-001";

    multimodalMultiImage(projectId, location, modelName);
  }

  // Generates content from multiple input images.
  public static void multimodalMultiImage(String projectId, String location, String modelName)
      throws IOException {
    // Initialize client that will be used to send requests. This client only needs
    // to be created once, and can be reused for multiple requests.
    try (VertexAI vertexAI = new VertexAI(projectId, location)) {
      GenerativeModel model = new GenerativeModel(modelName, vertexAI);

      Content content = ContentMaker.fromMultiModalData(
          PartMaker.fromMimeTypeAndData("image/png", readImageFile(
              "https://storage.googleapis.com/cloud-samples-data/vertex-ai/llm/prompts/landmark1.png")),
          "city: Rome, Landmark: the Colosseum",
          PartMaker.fromMimeTypeAndData("image/png", readImageFile(
              "https://storage.googleapis.com/cloud-samples-data/vertex-ai/llm/prompts/landmark2.png")),
          "city: Beijing, Landmark: Forbidden City",
          PartMaker.fromMimeTypeAndData("image/png", readImageFile(
              "https://storage.googleapis.com/cloud-samples-data/vertex-ai/llm/prompts/landmark3.png"))
      );

      GenerateContentResponse response = model.generateContent(content);

      String output = ResponseHandler.getText(response);
      System.out.println(output);
    }
  }

  // Reads the image data from the given URL.
  public static byte[] readImageFile(String url) throws IOException {
    URL urlObj = new URL(url);
    HttpURLConnection connection = (HttpURLConnection) urlObj.openConnection();
    connection.setRequestMethod("GET");

    int responseCode = connection.getResponseCode();

    if (responseCode == HttpURLConnection.HTTP_OK) {
      InputStream inputStream = connection.getInputStream();
      ByteArrayOutputStream outputStream = new ByteArrayOutputStream();

      byte[] buffer = new byte[1024];
      int bytesRead;
      while ((bytesRead = inputStream.read(buffer)) != -1) {
        outputStream.write(buffer, 0, bytesRead);
      }

      return outputStream.toByteArray();
    } else {
      throw new RuntimeException("Error fetching file: " + responseCode);
    }
  }
}

Go

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Go 设置说明执行操作。 如需了解详情，请参阅 Vertex AI Go API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。 如需了解详情，请参阅为本地开发环境设置身份验证。

import (
	"context"
	"fmt"
	"io"
	"log"
	"net/http"
	"net/url"
	"os"
	"strings"

	"cloud.google.com/go/vertexai/genai"
)

func main() {
	projectID := os.Getenv("GOOGLE_CLOUD_PROJECT")
	location := "us-central1"
	modelName := "gemini-1.5-flash-001"
	temperature := 0.4

	if projectID == "" {
		log.Fatal("require environment variable GOOGLE_CLOUD_PROJECT")
	}

	// construct this multimodal prompt:
	// [image of colosseum] city: Rome, Landmark: the Colosseum
	// [image of forbidden city]  city: Beijing, Landmark: the Forbidden City
	// [new image]

	// create prompt image parts
	// colosseum
	colosseum, err := partFromImageURL("https://storage.googleapis.com/cloud-samples-data/vertex-ai/llm/prompts/landmark1.png")
	if err != nil {
		log.Fatalf("unable to read image: %v", err)
	}
	// forbidden city
	forbiddenCity, err := partFromImageURL("https://storage.googleapis.com/cloud-samples-data/vertex-ai/llm/prompts/landmark2.png")
	if err != nil {
		log.Fatalf("unable to read image: %v", err)
	}
	// new image
	newImage, err := partFromImageURL("https://storage.googleapis.com/cloud-samples-data/vertex-ai/llm/prompts/landmark3.png")
	if err != nil {
		log.Fatalf("unable to read image: %v", err)
	}

	// create a multimodal (multipart) prompt
	prompt := []genai.Part{
		colosseum,
		genai.Text("city: Rome, Landmark: the Colosseum "),
		forbiddenCity,
		genai.Text("city: Beijing, Landmark: the Forbidden City "),
		newImage,
	}

	// generate the response
	err = generateMultimodalContent(os.Stdout, prompt, projectID, location, modelName, float32(temperature))
	if err != nil {
		log.Fatalf("unable to generate: %v", err)
	}
}

// generateMultimodalContent provide a generated response using multimodal input
func generateMultimodalContent(w io.Writer, parts []genai.Part, projectID, location, modelName string, temperature float32) error {
	ctx := context.Background()

	client, err := genai.NewClient(ctx, projectID, location)
	if err != nil {
		log.Fatal(err)
	}
	defer client.Close()

	model := client.GenerativeModel(modelName)
	model.SetTemperature(temperature)

	res, err := model.GenerateContent(ctx, parts...)
	if err != nil {
		return fmt.Errorf("unable to generate contents: %v", err)
	}

	fmt.Fprintf(w, "generated response: %s\n", res.Candidates[0].Content.Parts[0])

	return nil
}

// partFromImageURL create a multimodal prompt part from an image URL
func partFromImageURL(image string) (genai.Part, error) {
	var img genai.Blob

	imageURL, err := url.Parse(image)
	if err != nil {
		return img, err
	}
	res, err := http.Get(image)
	if err != nil || res.StatusCode != 200 {
		return img, err
	}
	defer res.Body.Close()
	data, err := io.ReadAll(res.Body)
	if err != nil {
		return img, fmt.Errorf("unable to read from http: %v", err)
	}

	position := strings.LastIndex(imageURL.Path, ".")
	if position == -1 {
		return img, fmt.Errorf("couldn't find a period to indicate a file extension")
	}
	ext := imageURL.Path[position+1:]

	img = genai.ImageData(ext, data)
	return img, nil
}

C#

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 C# 设置说明执行操作。 如需了解详情，请参阅 Vertex AI C# API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。 如需了解详情，请参阅为本地开发环境设置身份验证。

using Google.Api.Gax.Grpc;
using Google.Cloud.AIPlatform.V1;
using Google.Protobuf;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;

public class MultimodalMultiImage
{
    public async Task<string> GenerateContent(
        string projectId = "your-project-id",
        string location = "us-central1",
        string publisher = "google",
        string model = "gemini-1.5-flash-001"
    )
    {
        var predictionServiceClient = new PredictionServiceClientBuilder
        {
            Endpoint = $"{location}-aiplatform.googleapis.com"
        }.Build();

        ByteString colosseum = await ReadImageFileAsync(
            "https://storage.googleapis.com/cloud-samples-data/vertex-ai/llm/prompts/landmark1.png");

        ByteString forbiddenCity = await ReadImageFileAsync(
            "https://storage.googleapis.com/cloud-samples-data/vertex-ai/llm/prompts/landmark2.png");

        ByteString christRedeemer = await ReadImageFileAsync(
            "https://storage.googleapis.com/cloud-samples-data/vertex-ai/llm/prompts/landmark3.png");

        var generateContentRequest = new GenerateContentRequest
        {
            Model = $"projects/{projectId}/locations/{location}/publishers/{publisher}/models/{model}",
            Contents =
            {
                new Content
                {
                    Role = "USER",
                    Parts =
                    {
                        new Part { InlineData = new() { MimeType = "image/png", Data = colosseum }},
                        new Part { Text = "city: Rome, Landmark: the Colosseum" },
                        new Part { InlineData = new() { MimeType = "image/png", Data = forbiddenCity }},
                        new Part { Text = "city: Beijing, Landmark: Forbidden City"},
                        new Part { InlineData = new() { MimeType = "image/png", Data = christRedeemer }}
                    }
                }
            }
        };

        using PredictionServiceClient.StreamGenerateContentStream response = predictionServiceClient.StreamGenerateContent(generateContentRequest);

        StringBuilder fullText = new();

        AsyncResponseStream<GenerateContentResponse> responseStream = response.GetResponseStream();
        await foreach (GenerateContentResponse responseItem in responseStream)
        {
            fullText.Append(responseItem.Candidates[0].Content.Parts[0].Text);
        }
        return fullText.ToString();
    }

    private static async Task<ByteString> ReadImageFileAsync(string url)
    {
        using HttpClient client = new();
        using var response = await client.GetAsync(url);
        byte[] imageBytes = await response.Content.ReadAsByteArrayAsync();
        return ByteString.CopyFrom(imageBytes);
    }
}

函数

另请参阅函数调用。

REST

如需使用 Vertex AI API 测试函数提示，请向发布者模型端点发送 POST 请求。

在使用任何请求数据之前，请先进行以下替换：

• PROJECT_ID：您的项目 ID。

如需了解其他字段，请查看请求正文表。

HTTP 方法和网址：

POST https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-1.0-pro:streamGenerateContent

请求 JSON 正文：

{
  "contents": {
    "role": "user",
    "parts": {
      "text": "Which theaters in Mountain View show Barbie movie?"
    }
  },
  "tools": [
    {
      "function_declarations": [
        {
          "name": "find_movies",
          "description": "find movie titles currently playing in theaters based on any description, genre, title words, etc.",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA or a zip code e.g. 95616"
              },
              "description": {
                "type": "string",
                "description": "Any kind of description including category or genre, title words, attributes, etc."
              }
            },
            "required": [
              "description"
            ]
          }
        },
        {
          "name": "find_theaters",
          "description": "find theaters based on location and optionally movie title which are is currently playing in theaters",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA or a zip code e.g. 95616"
              },
              "movie": {
                "type": "string",
                "description": "Any movie title"
              }
            },
            "required": [
              "location"
            ]
          }
        },
        {
          "name": "get_showtimes",
          "description": "Find the start times for movies playing in a specific theater",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA or a zip code e.g. 95616"
              },
              "movie": {
                "type": "string",
                "description": "Any movie title"
              },
              "theater": {
                "type": "string",
                "description": "Name of the theater"
              },
              "date": {
                "type": "string",
                "description": "Date for requested showtime"
              }
            },
            "required": [
              "location",
              "movie",
              "theater",
              "date"
            ]
          }
        }
      ]
    }
  ]
}

如需发送请求，请选择以下方式之一：

curl

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI，或者使用了 Cloud Shell，这会使您自动登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

将请求正文保存在名为 request.json 的文件中，然后执行以下命令：

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-1.0-pro:streamGenerateContent"

PowerShell

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

将请求正文保存在名为 request.json 的文件中，然后执行以下命令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-1.0-pro:streamGenerateContent" | Select-Object -Expand Content

您应该会收到类似示例响应的 JSON 响应。

Python

如需了解如何安装或更新 Python 版 Vertex AI SDK，请参阅安装 Python 版 Vertex AI SDK。 如需了解详情，请参阅 Python API 参考文档。

import vertexai
from vertexai.generative_models import (
    Content,
    FunctionDeclaration,
    GenerationConfig,
    GenerativeModel,
    Part,
    Tool,
)

# Initialize Vertex AI
# TODO(developer): Update and un-comment below lines
# project_id = "PROJECT_ID"
vertexai.init(project=project_id, location="us-central1")

# Initialize Gemini model
model = GenerativeModel(model_name="gemini-1.0-pro-001")

# Define the user's prompt in a Content object that we can reuse in model calls
user_prompt_content = Content(
    role="user",
    parts=[
        Part.from_text("What is the weather like in Boston?"),
    ],
)

# Specify a function declaration and parameters for an API request
function_name = "get_current_weather"
get_current_weather_func = FunctionDeclaration(
    name=function_name,
    description="Get the current weather in a given location",
    # Function parameters are specified in OpenAPI JSON schema format
    parameters={
        "type": "object",
        "properties": {"location": {"type": "string", "description": "Location"}},
    },
)

# Define a tool that includes the above get_current_weather_func
weather_tool = Tool(
    function_declarations=[get_current_weather_func],
)

# Send the prompt and instruct the model to generate content using the Tool that you just created
response = model.generate_content(
    user_prompt_content,
    generation_config=GenerationConfig(temperature=0),
    tools=[weather_tool],
)
function_call = response.candidates[0].function_calls[0]
print(function_call)

# Check the function name that the model responded with, and make an API call to an external system
if function_call.name == function_name:
    # Extract the arguments to use in your API call
    location = function_call.args["location"]  # noqa: F841

    # Here you can use your preferred method to make an API request to fetch the current weather, for example:
    # api_response = requests.post(weather_api_url, data={"location": location})

    # In this example, we'll use synthetic data to simulate a response payload from an external API
    api_response = """{ "location": "Boston, MA", "temperature": 38, "description": "Partly Cloudy",
                    "icon": "partly-cloudy", "humidity": 65, "wind": { "speed": 10, "direction": "NW" } }"""

# Return the API response to Gemini so it can generate a model response or request another function call
response = model.generate_content(
    [
        user_prompt_content,  # User prompt
        response.candidates[0].content,  # Function call response
        Content(
            parts=[
                Part.from_function_response(
                    name=function_name,
                    response={
                        "content": api_response,  # Return the API response to Gemini
                    },
                ),
            ],
        ),
    ],
    tools=[weather_tool],
)

# Get the model response
print(response.text)

响应示例

文本

data: {"candidates": [{"content": {"role": "model","parts": [{"text": "Ingredients:\n\n- 3 ripe bananas, mashed\n- 1 cup sugar"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}]}]}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": "\n- 1/2 cup (1 stick) unsalted butter, softened\n"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}]}]}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": "- 2 large eggs\n- 2 cups all-purpose flour\n- 1 teaspoon baking soda\n- 1/2 teaspoon salt\n- "}]},"safetyRatings": [{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}]}]}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": "1/2 cup chopped walnuts (optional)\n\nInstructions:\n\n1. Preheat oven to 350 degrees F (175 degrees C). Grease"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}]}]}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": " and flour a 9x5 inch loaf pan.\n2. In a large bowl, cream together the butter and sugar until light and fluffy. Beat in the eggs one at a time, then stir in the mashed bananas.\n3"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}],"citationMetadata": {"citations": [{"startIndex": 322,"endIndex": 451,"uri": "https://discover.texasrealfood.com/texas-home-cooking/whats-in-season-plums-exploring-health-benefits-varieties-and-recipes"}]}}]}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": ". In a separate bowl, whisk together the flour, baking soda, and salt. Gradually add the dry ingredients to the wet ingredients, mixing until just combined. Fold in the walnuts, if desired.\n4. Pour the batter into the"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}],"citationMetadata": {"citations": [{"startIndex": 472,"endIndex": 614,"uri": "https://commandame.com/urban-cookhouse-half-baked-cookie-recipe/"}]}}]}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": " prepared loaf pan and bake for 50-60 minutes, or until a toothpick inserted into the center comes out clean.\n5. Let the bread cool in the pan for 10 minutes before turning it out onto a wire rack to cool completely."}]},"finishReason": "STOP","safetyRatings": [{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}],"citationMetadata": {"citations": [{"startIndex": 666,"endIndex": 796,"uri": "https://dessertdonelight.com/healthy-hawaiian-banana-bread/"},{"startIndex": 728,"endIndex": 851,"uri": "https://earlsgrocery.com/gluten-free-bread/gluten-free-yeast-free-bread-healthy-and-nutition"}]}}],"usageMetadata": {"promptTokenCount": 8,"candidatesTokenCount": 245,"totalTokenCount": 253}}

聊天

data: {"candidates": [{"content": {"role": "model","parts": [{"text": "Avast there, landlubber! Ye be mistaken. I be but a"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HARASSMENT","probability": "LOW"},{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}]}]}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": " humble pirate of the seven seas, brought to life by the magic of artificial intelligence"}]},"safetyRatings": [{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}]}]}

data: {"candidates": [{"content": {"role": "model","parts": [{"text": ". I be no real-life pirate, but I be mighty good at pretendin'!"}]},"finishReason": "STOP","safetyRatings": [{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}]}],"usageMetadata": {"promptTokenCount": 23,"candidatesTokenCount": 50,"totalTokenCount": 73}}

多模态

[{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": " A daisy is growing up through a pile of brown and yellow fall leaves"
          }
        ]
      },
      "finishReason": "STOP",
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE"
        },
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE"
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE"
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE"
        }
      ]
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 262,
    "candidatesTokenCount": 14,
    "totalTokenCount": 276
  }
}]

函数

[{
  "candidates": [
    {
      "content": {
        "parts": [
          {
            "functionCall": {
              "name": "find_theaters",
              "args": {
                "movie": "Barbie",
                "location": "Mountain View, CA"
              }
            }
          }
        ]
      },
      "finishReason": "STOP",
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE"
        },
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE"
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE"
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE"
        }
      ]
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 9,
    "totalTokenCount": 9
  }
}]

后续步骤

了解如何使用 Vertex AI API for Gemini：

• 了解如何发送聊天提示请求。
• 了解如何发送多模态提示请求。
• 了解如何调用函数。
• 了解如何获取词元数。
• 了解如何配置安全属性。