Package Classes (1.74.0)

Summary of entries of Classes for aiplatform.

Classes

CustomMetric

The custom evaluation metric.

A fully-customized CustomMetric that can be used to evaluate a single model by defining a metric function for a computation-based metric. The CustomMetric is computed on the client-side using the user-defined metric function in SDK only, not by the Vertex Gen AI Evaluation Service.

Attributes: name: The name of the metric. metric_function: The user-defined evaluation function to compute a metric score. Must use the dataset row dictionary as the metric function input and return per-instance metric result as a dictionary output. The metric score must mapped to the name of the CustomMetric as key.

EvalResult

Evaluation result.

EvalTask

A class representing an EvalTask.

An Evaluation Tasks is defined to measure the model's ability to perform a certain task in response to specific prompts or inputs. Evaluation tasks must contain an evaluation dataset, and a list of metrics to evaluate. Evaluation tasks help developers compare propmpt templates, track experiments, compare models and their settings, and assess the quality of the model's generated text.

Dataset Details:

Default dataset column names:
    * prompt_column_name: "prompt"
    * reference_column_name: "reference"
    * response_column_name: "response"
    * baseline_model_response_column_name: "baseline_model_response"

Requirement for different use cases:
  * Bring-your-own-response (BYOR): You already have the data that you
      want to evaluate stored in the dataset. Response column name can be
      customized by providing `response_column_name` parameter, or in the
      `metric_column_mapping`. For BYOR pairwise evaluation, the baseline
      model response column name can be customized by providing
      `baseline_model_response_column_name` parameter, or
      in the `metric_column_mapping`. If the `response` column or
      `baseline_model_response` column is present while the
      corresponding model is specified, an error will be raised.

  * Perform model inference without a prompt template: You have a dataset
      containing the input prompts to the model and want to perform model
      inference before evaluation. A column named `prompt` is required
      in the evaluation dataset and is used directly as input to the model.

  * Perform model inference with a prompt template: You have a dataset
      containing the input variables to the prompt template and want to
      assemble the prompts for model inference. Evaluation dataset
      must contain column names corresponding to the variable names in
      the prompt template. For example, if prompt template is
      "Instruction: {instruction}, context: {context}", the dataset must
      contain `instruction` and `context` columns.

Metrics Details:

The supported metrics descriptions, rating rubrics, and the required
input variables can be found on the Vertex AI public documentation page.
[Evaluation methods and metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval).

Usage Examples:

1. To perform bring-your-own-response(BYOR) evaluation, provide the model
responses in the `response` column in the dataset. If a pairwise metric is
used for BYOR evaluation, provide the baseline model responses in the
`baseline_model_response` column.

  ```
  eval_dataset = pd.DataFrame({
          "prompt"  : [...],
          "reference": [...],
          "response" : [...],
          "baseline_model_response": [...],
  })
  eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=[
            "bleu",
            "rouge_l_sum",
            MetricPromptTemplateExamples.Pointwise.FLUENCY,
            MetricPromptTemplateExamples.Pairwise.SAFETY
    ],
    experiment="my-experiment",
  )
  eval_result = eval_task.evaluate(experiment_run_name="eval-experiment-run")
  ```

2. To perform evaluation with Gemini model inference, specify the `model`
parameter with a `GenerativeModel` instance.  The input column name to the
model is `prompt` and must be present in the dataset.

  ```
  eval_dataset = pd.DataFrame({
        "reference": [...],
        "prompt"  : [...],
  })
  result = EvalTask(
      dataset=eval_dataset,
      metrics=["exact_match", "bleu", "rouge_1", "rouge_l_sum"],
      experiment="my-experiment",
  ).evaluate(
      model=GenerativeModel("gemini-1.5-pro"),
      experiment_run_name="gemini-eval-run"
  )
  ```

3. If a `prompt_template` is specified, the `prompt` column is not required.
Prompts can be assembled from the evaluation dataset, and all prompt
template variable names must be present in the dataset columns.
  ```
  eval_dataset = pd.DataFrame({
      "context"    : [...],
      "instruction": [...],
  })
  result = EvalTask(
      dataset=eval_dataset,
      metrics=[MetricPromptTemplateExamples.Pointwise.SUMMARIZATION_QUALITY],
  ).evaluate(
      model=GenerativeModel("gemini-1.5-pro"),
      prompt_template="{instruction}. Article: {context}. Summary:",
  )
  ```

4. To perform evaluation with custom model inference, specify the `model`
parameter with a custom inference function. The input column name to the
custom inference function is `prompt` and must be present in the dataset.

  ```
  from openai import OpenAI
  client = OpenAI()
  def custom_model_fn(input: str) -> str:
    response = client.chat.completions.create(
      model="gpt-3.5-turbo",
      messages=[
        {"role": "user", "content": input}
      ]
    )
    return response.choices[0].message.content

  eval_dataset = pd.DataFrame({
        "prompt"  : [...],
        "reference": [...],
  })
  result = EvalTask(
      dataset=eval_dataset,
      metrics=[MetricPromptTemplateExamples.Pointwise.SAFETY],
      experiment="my-experiment",
  ).evaluate(
      model=custom_model_fn,
      experiment_run_name="gpt-eval-run"
  )
  ```

5. To perform pairwise metric evaluation with model inference step, specify
the `baseline_model` input to a `PairwiseMetric` instance and the candidate
`model` input to the `EvalTask.evaluate()` function. The input column name
to both models is `prompt` and must be present in the dataset.

  ```
  baseline_model = GenerativeModel("gemini-1.0-pro")
  candidate_model = GenerativeModel("gemini-1.5-pro")

  pairwise_groundedness = PairwiseMetric(
      metric_prompt_template=MetricPromptTemplateExamples.get_prompt_template(
          "pairwise_groundedness"
      ),
      baseline_model=baseline_model,
  )
  eval_dataset = pd.DataFrame({
        "prompt"  : [...],
  })
  result = EvalTask(
      dataset=eval_dataset,
      metrics=[pairwise_groundedness],
      experiment="my-pairwise-experiment",
  ).evaluate(
      model=candidate_model,
      experiment_run_name="gemini-pairwise-eval-run",
  )
  ```

MetricPromptTemplateExamples

Examples of metric prompt templates for model-based evaluation.

Pairwise

Example PairwiseMetric instances.

Pointwise

Example PointwiseMetric instances.

PairwiseMetric

A Model-based Pairwise Metric.

A model-based evaluation metric that compares two generative models' responses side-by-side, and allows users to A/B test their generative models to determine which model is performing better.

For more details on when to use pairwise metrics, see Evaluation methods and metrics.

Result Details:

* In `EvalResult.summary_metrics`, win rates for both the baseline and
candidate model are computed. The win rate is computed as proportion of
wins of one model's responses to total attempts as a decimal value
between 0 and 1.

* In `EvalResult.metrics_table`, a pairwise metric produces two
evaluation results per dataset row:
    * `pairwise_choice`: The choice shows whether the candidate model or
      the baseline model performs better, or if they are equally good.
    * `explanation`: The rationale behind each verdict using
      chain-of-thought reasoning. The explanation helps users scrutinize
      the judgment and builds appropriate trust in the decisions.

See [documentation
page](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval#understand-results)
for more details on understanding the metric results.

Usage Examples:

```
baseline_model = GenerativeModel("gemini-1.0-pro")
candidate_model = GenerativeModel("gemini-1.5-pro")

pairwise_groundedness = PairwiseMetric(
    metric_prompt_template=MetricPromptTemplateExamples.get_prompt_template(
        "pairwise_groundedness"
    ),
    baseline_model=baseline_model,
)
eval_dataset = pd.DataFrame({
      "prompt"  : [...],
})
pairwise_task = EvalTask(
    dataset=eval_dataset,
    metrics=[pairwise_groundedness],
    experiment="my-pairwise-experiment",
)
pairwise_result = pairwise_task.evaluate(
    model=candidate_model,
    experiment_run_name="gemini-pairwise-eval-run",
)
```

PairwiseMetricPromptTemplate

Pairwise metric prompt template for pairwise model-based metrics.

PointwiseMetric

A Model-based Pointwise Metric.

A model-based evaluation metric that evaluate a single generative model's response.

For more details on when to use model-based pointwise metrics, see Evaluation methods and metrics.

Usage Examples:

```
candidate_model = GenerativeModel("gemini-1.5-pro")
eval_dataset = pd.DataFrame({
    "prompt"  : [...],
})
fluency_metric = PointwiseMetric(
    metric="fluency",
    metric_prompt_template=MetricPromptTemplateExamples.get_prompt_template('fluency'),
)
pointwise_eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=[
        fluency_metric,
        MetricPromptTemplateExamples.Pointwise.GROUNDEDNESS,
    ],
)
pointwise_result = pointwise_eval_task.evaluate(
    model=candidate_model,
)
```

PointwiseMetricPromptTemplate

Pointwise metric prompt template for pointwise model-based metrics.

PromptTemplate

A prompt template for creating prompts with variables.

The PromptTemplate class allows users to define a template string with variables represented in curly braces {variable}. The variable names cannot contain spaces. These variables can be replaced with specific values using the assemble method, providing flexibility in generating dynamic prompts.

Usage:

```
template_str = "Hello, {name}! Today is {day}. How are you?"
prompt_template = PromptTemplate(template_str)
completed_prompt = prompt_template.assemble(name="John", day="Monday")
print(completed_prompt)
```

Rouge

The ROUGE Metric.

Calculates the recall of n-grams in prediction as compared to reference and returns a score ranging between 0 and 1. Supported rouge types are rougen[1-9], rougeL, and rougeLsum.

Candidate

A response candidate generated by the model.

ChatSession

Chat session holds the chat history.

Content

The multi-part content of a message.

Usage:

response = model.generate_content(contents=[
    Content(role="user", parts=[Part.from_text("Why is sky blue?")])
])
```

FinishReason

The reason why the model stopped generating tokens. If empty, the model has not stopped generating the tokens.

FunctionCall

Function call.

FunctionDeclaration

A representation of a function declaration.

Usage: Create function declaration and tool:

get_current_weather_func = generative_models.FunctionDeclaration(
    name="get_current_weather",
    description="Get the current weather in a given location",
    parameters={
        "type": "object",
        "properties": {
            "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
            },
            "unit": {
                "type": "string",
                "enum": [
                    "celsius",
                    "fahrenheit",
                ]
            }
        },
        "required": [
            "location"
        ]
    },
    # Optional:
    response={
        "type": "object",
        "properties": {
            "weather": {
                "type": "string",
                "description": "The weather in the city"
            },
        },
    },
)
weather_tool = generative_models.Tool(
    function_declarations=[get_current_weather_func],
)
```
Use tool in `GenerativeModel.generate_content`:
```
model = GenerativeModel("gemini-pro")
print(model.generate_content(
    "What is the weather like in Boston?",
    # You can specify tools when creating a model to avoid having to send them with every request.
    tools=[weather_tool],
))
```
Use tool in chat:
```
model = GenerativeModel(
    "gemini-pro",
    # You can specify tools when creating a model to avoid having to send them with every request.
    tools=[weather_tool],
)
chat = model.start_chat()
print(chat.send_message("What is the weather like in Boston?"))
print(chat.send_message(
    Part.from_function_response(
        name="get_current_weather",
        response={
            "content": {"weather_there": "super nice"},
        }
    ),
))
```

GenerationConfig

Parameters for the generation.

RoutingConfig

The configuration for model router requests.

The routing config is either one of the two nested classes:

  • AutoRoutingMode: Automated routing.
  • ManualRoutingMode: Manual routing.

Usage:

  • AutoRoutingMode:

    routing_config=generative_models.RoutingConfig(
        routing_config=generative_models.RoutingConfig.AutoRoutingMode(
            model_routing_preference=generative_models.RoutingConfig.AutoRoutingMode.ModelRoutingPreference.BALANCED,
        ),
    )
    
  • ManualRoutingMode:

    routing_config=generative_models.RoutingConfig(
        routing_config=generative_models.RoutingConfig.ManutalRoutingMode(
            model_name="gemini-1.5-pro-001",
        ),
    )
    

AutoRoutingMode

When automated routing is specified, the routing will be determined by the routing model predicted quality and customer provided model routing preference.

ModelRoutingPreference

The model routing preference.

ManualRoutingMode

When manual routing is set, the specified model will be used directly.

GenerationResponse

The response from the model.

GenerativeModel

Initializes GenerativeModel.

Usage:

model = GenerativeModel("gemini-pro")
print(model.generate_content("Hello"))
```

HarmBlockThreshold

Probability based thresholds levels for blocking.

HarmCategory

Harm categories that will block the content.

Image

The image that can be sent to a generative model.

Part

A part of a multi-part Content message.

Usage:

text_part = Part.from_text("Why is sky blue?")
image_part = Part.from_image(Image.load_from_file("image.jpg"))
video_part = Part.from_uri(uri="gs://.../video.mp4", mime_type="video/mp4")
function_response_part = Part.from_function_response(
    name="get_current_weather",
    response={
        "content": {"weather_there": "super nice"},
    }
)

response1 = model.generate_content([text_part, image_part])
response2 = model.generate_content(video_part)
response3 = chat.send_message(function_response_part)
```

ResponseValidationError

API documentation for ResponseValidationError class.

SafetySetting

Parameters for the generation.

HarmBlockMethod

Probability vs severity.

HarmBlockThreshold

Probability based thresholds levels for blocking.

HarmCategory

Harm categories that will block the content.

Tool

A collection of functions that the model may use to generate response.

Usage: Create tool from function declarations:

get_current_weather_func = generative_models.FunctionDeclaration(...)
weather_tool = generative_models.Tool(
    function_declarations=[get_current_weather_func],
)
```
Use tool in `GenerativeModel.generate_content`:
```
model = GenerativeModel("gemini-pro")
print(model.generate_content(
    "What is the weather like in Boston?",
    # You can specify tools when creating a model to avoid having to send them with every request.
    tools=[weather_tool],
))
```
Use tool in chat:
```
model = GenerativeModel(
    "gemini-pro",
    # You can specify tools when creating a model to avoid having to send them with every request.
    tools=[weather_tool],
)
chat = model.start_chat()
print(chat.send_message("What is the weather like in Boston?"))
print(chat.send_message(
    Part.from_function_response(
        name="get_current_weather",
        response={
            "content": {"weather_there": "super nice"},
        }
    ),
))
```

ToolConfig

Config shared for all tools provided in the request.

Usage: Create ToolConfig

tool_config = ToolConfig(
    function_calling_config=ToolConfig.FunctionCallingConfig(
        mode=ToolConfig.FunctionCallingConfig.Mode.ANY,
        allowed_function_names=["get_current_weather_func"],
))
```
Use ToolConfig in `GenerativeModel.generate_content`:
```
model = GenerativeModel("gemini-pro")
print(model.generate_content(
    "What is the weather like in Boston?",
    # You can specify tools when creating a model to avoid having to send them with every request.
    tools=[weather_tool],
    tool_config=tool_config,
))
```
Use ToolConfig in chat:
```
model = GenerativeModel(
    "gemini-pro",
    # You can specify tools when creating a model to avoid having to send them with every request.
    tools=[weather_tool],
    tool_config=tool_config,
)
chat = model.start_chat()
print(chat.send_message("What is the weather like in Boston?"))
print(chat.send_message(
    Part.from_function_response(
        name="get_current_weather",
        response={
            "content": {"weather_there": "super nice"},
        }
    ),
))
```

grounding

Grounding namespace.

DynamicRetrievalConfig

Config for dynamic retrieval.

Mode

The mode of the predictor to be used in dynamic retrieval.

GoogleSearchRetrieval

Tool to retrieve public web data for grounding, powered by Google Search.

Retrieval

Defines a retrieval tool that model can call to access external knowledge.

VertexAISearch

Retrieve from Vertex AI Search data store for grounding. See https://cloud.google.com/products/agent-builder

ChatMessage

A chat message.

ChatModel

ChatModel represents a language model that is capable of chat.

Examples::

chat_model = ChatModel.from_pretrained("chat-bison@001")

chat = chat_model.start_chat(
    context="My name is Ned. You are my personal assistant. My favorite movies are Lord of the Rings and Hobbit.",
    examples=[
        InputOutputTextPair(
            input_text="Who do you work for?",
            output_text="I work for Ned.",
        ),
        InputOutputTextPair(
            input_text="What do I like?",
            output_text="Ned likes watching movies.",
        ),
    ],
    temperature=0.3,
)

chat.send_message("Do you know any cool events this weekend?")

ChatSession

ChatSession represents a chat session with a language model.

Within a chat session, the model keeps context and remembers the previous conversation.

CodeChatModel

CodeChatModel represents a model that is capable of completing code.

.. rubric:: Examples

code_chat_model = CodeChatModel.from_pretrained("codechat-bison@001")

code_chat = code_chat_model.start_chat( context="I'm writing a large-scale enterprise application.", max_output_tokens=128, temperature=0.2, )

code_chat.send_message("Please help write a function to calculate the min of two numbers")

CodeChatSession

CodeChatSession represents a chat session with code chat language model.

Within a code chat session, the model keeps context and remembers the previous converstion.

CodeGenerationModel

Creates a LanguageModel.

This constructor should not be called directly. Use LanguageModel.from_pretrained(model_name=...) instead.

GroundingSource

API documentation for GroundingSource class.

InlineContext

InlineContext represents a grounding source using provided inline context. .. attribute:: inline_context

The content used as inline context.

:type: str

VertexAISearch

VertexAISearchDatastore represents a grounding source using Vertex AI Search datastore .. attribute:: data_store_id

Data store ID of the Vertex AI Search datastore.

:type: str

WebSearch

WebSearch represents a grounding source using public web search. .. attribute:: disable_attribution

If set to True, skip finding claim attributions (i.e not generate grounding citation). Default: False.

:type: bool

InputOutputTextPair

InputOutputTextPair represents a pair of input and output texts.

TextEmbedding

Text embedding vector and statistics.

TextEmbeddingInput

Structural text embedding input.

TextEmbeddingModel

Creates a LanguageModel.

This constructor should not be called directly. Use LanguageModel.from_pretrained(model_name=...) instead.

TextGenerationModel

Creates a LanguageModel.

This constructor should not be called directly. Use LanguageModel.from_pretrained(model_name=...) instead.

TextGenerationResponse

TextGenerationResponse represents a response of a language model. .. attribute:: text

The generated text

:type: str

_TunableModelMixin

Model that can be tuned with supervised fine tuning (SFT).

AutomaticFunctionCallingResponder

Responder that automatically responds to model's function calls.

CallableFunctionDeclaration

A function declaration plus a function.

Candidate

A response candidate generated by the model.

ChatSession

Chat session holds the chat history.

Content

The multi-part content of a message.

Usage:

response = model.generate_content(contents=[
    Content(role="user", parts=[Part.from_text("Why is sky blue?")])
])
```

FinishReason

The reason why the model stopped generating tokens. If empty, the model has not stopped generating the tokens.

FunctionCall

Function call.

FunctionDeclaration

A representation of a function declaration.

Usage: Create function declaration and tool:

get_current_weather_func = generative_models.FunctionDeclaration(
    name="get_current_weather",
    description="Get the current weather in a given location",
    parameters={
        "type": "object",
        "properties": {
            "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
            },
            "unit": {
                "type": "string",
                "enum": [
                    "celsius",
                    "fahrenheit",
                ]
            }
        },
        "required": [
            "location"
        ]
    },
    # Optional:
    response={
        "type": "object",
        "properties": {
            "weather": {
                "type": "string",
                "description": "The weather in the city"
            },
        },
    },
)
weather_tool = generative_models.Tool(
    function_declarations=[get_current_weather_func],
)
```
Use tool in `GenerativeModel.generate_content`:
```
model = GenerativeModel("gemini-pro")
print(model.generate_content(
    "What is the weather like in Boston?",
    # You can specify tools when creating a model to avoid having to send them with every request.
    tools=[weather_tool],
))
```
Use tool in chat:
```
model = GenerativeModel(
    "gemini-pro",
    # You can specify tools when creating a model to avoid having to send them with every request.
    tools=[weather_tool],
)
chat = model.start_chat()
print(chat.send_message("What is the weather like in Boston?"))
print(chat.send_message(
    Part.from_function_response(
        name="get_current_weather",
        response={
            "content": {"weather_there": "super nice"},
        }
    ),
))
```

GenerationConfig

Parameters for the generation.

RoutingConfig

The configuration for model router requests.

The routing config is either one of the two nested classes:

  • AutoRoutingMode: Automated routing.
  • ManualRoutingMode: Manual routing.

Usage:

  • AutoRoutingMode:

    routing_config=generative_models.RoutingConfig(
        routing_config=generative_models.RoutingConfig.AutoRoutingMode(
            model_routing_preference=generative_models.RoutingConfig.AutoRoutingMode.ModelRoutingPreference.BALANCED,
        ),
    )
    
  • ManualRoutingMode:

    routing_config=generative_models.RoutingConfig(
        routing_config=generative_models.RoutingConfig.ManutalRoutingMode(
            model_name="gemini-1.5-pro-001",
        ),
    )
    

AutoRoutingMode

When automated routing is specified, the routing will be determined by the routing model predicted quality and customer provided model routing preference.

ModelRoutingPreference

The model routing preference.

ManualRoutingMode

When manual routing is set, the specified model will be used directly.

GenerationResponse

The response from the model.

GenerativeModel

Initializes GenerativeModel.

Usage:

model = GenerativeModel("gemini-pro")
print(model.generate_content("Hello"))
```

HarmBlockThreshold

Probability based thresholds levels for blocking.

HarmCategory

Harm categories that will block the content.

Image

The image that can be sent to a generative model.

Part

A part of a multi-part Content message.

Usage:

text_part = Part.from_text("Why is sky blue?")
image_part = Part.from_image(Image.load_from_file("image.jpg"))
video_part = Part.from_uri(uri="gs://.../video.mp4", mime_type="video/mp4")
function_response_part = Part.from_function_response(
    name="get_current_weather",
    response={
        "content": {"weather_there": "super nice"},
    }
)

response1 = model.generate_content([text_part, image_part])
response2 = model.generate_content(video_part)
response3 = chat.send_message(function_response_part)
```

ResponseBlockedError

API documentation for ResponseBlockedError class.

ResponseValidationError

API documentation for ResponseValidationError class.

SafetySetting

Parameters for the generation.

HarmBlockMethod

Probability vs severity.

HarmBlockThreshold

Probability based thresholds levels for blocking.

HarmCategory

Harm categories that will block the content.

Tool

A collection of functions that the model may use to generate response.

Usage: Create tool from function declarations:

get_current_weather_func = generative_models.FunctionDeclaration(...)
weather_tool = generative_models.Tool(
    function_declarations=[get_current_weather_func],
)
```
Use tool in `GenerativeModel.generate_content`:
```
model = GenerativeModel("gemini-pro")
print(model.generate_content(
    "What is the weather like in Boston?",
    # You can specify tools when creating a model to avoid having to send them with every request.
    tools=[weather_tool],
))
```
Use tool in chat:
```
model = GenerativeModel(
    "gemini-pro",
    # You can specify tools when creating a model to avoid having to send them with every request.
    tools=[weather_tool],
)
chat = model.start_chat()
print(chat.send_message("What is the weather like in Boston?"))
print(chat.send_message(
    Part.from_function_response(
        name="get_current_weather",
        response={
            "content": {"weather_there": "super nice"},
        }
    ),
))
```

ToolConfig

Config shared for all tools provided in the request.

Usage: Create ToolConfig

tool_config = ToolConfig(
    function_calling_config=ToolConfig.FunctionCallingConfig(
        mode=ToolConfig.FunctionCallingConfig.Mode.ANY,
        allowed_function_names=["get_current_weather_func"],
))
```
Use ToolConfig in `GenerativeModel.generate_content`:
```
model = GenerativeModel("gemini-pro")
print(model.generate_content(
    "What is the weather like in Boston?",
    # You can specify tools when creating a model to avoid having to send them with every request.
    tools=[weather_tool],
    tool_config=tool_config,
))
```
Use ToolConfig in chat:
```
model = GenerativeModel(
    "gemini-pro",
    # You can specify tools when creating a model to avoid having to send them with every request.
    tools=[weather_tool],
    tool_config=tool_config,
)
chat = model.start_chat()
print(chat.send_message("What is the weather like in Boston?"))
print(chat.send_message(
    Part.from_function_response(
        name="get_current_weather",
        response={
            "content": {"weather_there": "super nice"},
        }
    ),
))
```

ChatMessage

A chat message.

CountTokensResponse

The response from a count_tokens request. .. attribute:: total_tokens

The total number of tokens counted across all instances passed to the request.

:type: int

EvaluationClassificationMetric

The evaluation metric response for classification metrics.

EvaluationMetric

The evaluation metric response.

EvaluationQuestionAnsweringSpec

Spec for question answering model evaluation tasks.

EvaluationTextClassificationSpec

Spec for text classification model evaluation tasks.

EvaluationTextGenerationSpec

Spec for text generation model evaluation tasks.

EvaluationTextSummarizationSpec

Spec for text summarization model evaluation tasks.

InputOutputTextPair

InputOutputTextPair represents a pair of input and output texts.

TextEmbedding

Text embedding vector and statistics.

TextEmbeddingInput

Structural text embedding input.

TextGenerationResponse

TextGenerationResponse represents a response of a language model. .. attribute:: text

The generated text

:type: str

TuningEvaluationSpec

Specification for model evaluation to perform during tuning.

LangchainAgent

A Langchain Agent.

See https://cloud.google.com/vertex-ai/generative-ai/docs/reasoning-engine/develop for details.

Queryable

Protocol for Reasoning Engine applications that can be queried.

ReasoningEngine

Represents a Vertex AI Reasoning Engine resource.

TuningJob

Represents a TuningJob that runs with Google owned models.

SupervisedTuningJob

Initializes class with project, location, and api_client.

ControlImageConfig

Control image config.

ControlReferenceImage

Control reference image.

This encapsulates the control reference image type.

EntityLabel

Entity label holding a text label and any associated confidence score.

GeneratedImage

Generated image.

GeneratedMask

Generated image mask.

Image

Image.

ImageCaptioningModel

Generates captions from image.

Examples::

model = ImageCaptioningModel.from_pretrained("imagetext@001")
image = Image.load_from_file("image.png")
captions = model.get_captions(
    image=image,
    # Optional:
    number_of_results=1,
    language="en",
)

ImageGenerationModel

Generates images from text prompt.

Examples::

model = ImageGenerationModel.from_pretrained("imagegeneration@002")
response = model.generate_images(
    prompt="Astronaut riding a horse",
    # Optional:
    number_of_images=1,
    seed=0,
)
response[0].show()
response[0].save("image1.png")

ImageGenerationResponse

Image generation response.

ImageQnAModel

Answers questions about an image.

Examples::

model = ImageQnAModel.from_pretrained("imagetext@001")
image = Image.load_from_file("image.png")
answers = model.ask_question(
    image=image,
    question="What color is the car in this image?",
    # Optional:
    number_of_results=1,
)

ImageSegmentationModel

Segments an image.

ImageSegmentationResponse

Image Segmentation response.

ImageTextModel

Generates text from images.

Examples::

model = ImageTextModel.from_pretrained("imagetext@001")
image = Image.load_from_file("image.png")

captions = model.get_captions(
    image=image,
    # Optional:
    number_of_results=1,
    language="en",
)

answers = model.ask_question(
    image=image,
    question="What color is the car in this image?",
    # Optional:
    number_of_results=1,
)

MaskImageConfig

Mask image config.

MaskReferenceImage

Mask reference image. This encapsulates the mask reference image type.

MultiModalEmbeddingModel

Generates embedding vectors from images and videos.

Examples::

model = MultiModalEmbeddingModel.from_pretrained("multimodalembedding@001")
image = Image.load_from_file("image.png")
video = Video.load_from_file("video.mp4")

embeddings = model.get_embeddings(
    image=image,
    video=video,
    contextual_text="Hello world",
)
image_embedding = embeddings.image_embedding
video_embeddings = embeddings.video_embeddings
text_embedding = embeddings.text_embedding

MultiModalEmbeddingResponse

The multimodal embedding response.

RawReferenceImage

Raw reference image.

This encapsulates the raw reference image type.

ReferenceImage

Reference image.

This is a new base API object for Imagen 3.0 Capabilities.

Scribble

Input scribble for image segmentation.

StyleImageConfig

Style image config.

StyleReferenceImage

Style reference image. This encapsulates the style reference image type.

SubjectImageConfig

Subject image config.

SubjectReferenceImage

Subject reference image.

This encapsulates the subject reference image type.

Video

Video.

VideoEmbedding

Embeddings generated from video with offset times.

VideoSegmentConfig

The specific video segments (in seconds) the embeddings are generated for.

WatermarkVerificationModel

Verifies if an image has a watermark.

WatermarkVerificationResponse

WatermarkVerificationResponse(_prediction_response: Any, watermark_verification_result: Optional[str] = None)

ModelMonitor

Initializer for ModelMonitor.

ModelMonitoringJob

Initializer for ModelMonitoringJob.

Example Usage:

 my_monitoring_job = aiplatform.ModelMonitoringJob(
     model_monitoring_job_name='projects/123/locations/us-central1/modelMonitors/\
     my_model_monitor_id/modelMonitoringJobs/my_monitoring_job_id'
 )
 or
 my_monitoring_job = aiplatform.aiplatform.ModelMonitoringJob(
     model_monitoring_job_name='my_monitoring_job_id',
     model_monitor_id='my_model_monitor_id',
 )

DataDriftSpec

Data drift monitoring spec.

Data drift measures the distribution distance between the current dataset and a baseline dataset. A typical use case is to detect data drift between the recent production serving dataset and the training dataset, or to compare the recent production dataset with a dataset from a previous period.

.. rubric:: Example

feature_drift_spec=DataDriftSpec( features=["feature1"] categorical_metric_type="l_infinity", numeric_metric_type="jensen_shannon_divergence", default_categorical_alert_threshold=0.01, default_numeric_alert_threshold=0.02, feature_alert_thresholds={"feature1":0.02, "feature2":0.01}, )

FeatureAttributionSpec

Feature attribution spec.

.. rubric:: Example

feature_attribution_spec=FeatureAttributionSpec( features=["feature1"] default_alert_threshold=0.01, feature_alert_thresholds={"feature1":0.02, "feature2":0.01}, batch_dedicated_resources=BatchDedicatedResources( starting_replica_count=1, max_replica_count=2, machine_spec=my_machine_spec, ), )

FieldSchema

Field Schema.

The class identifies the data type of a single feature, which combines together to form the Schema for different fields in ModelMonitoringSchema.

ModelMonitoringSchema

Initializer for ModelMonitoringSchema.

MonitoringInput

Model monitoring data input spec.

NotificationSpec

Initializer for NotificationSpec.

ObjectiveSpec

Initializer for ObjectiveSpec.

OutputSpec

Initializer for OutputSpec.

TabularObjective

Initializer for TabularObjective.

GeneratedImage

Generated image.

Image

Image.

ImageCaptioningModel

Generates captions from image.

Examples::

model = ImageCaptioningModel.from_pretrained("imagetext@001")
image = Image.load_from_file("image.png")
captions = model.get_captions(
    image=image,
    # Optional:
    number_of_results=1,
    language="en",
)

ImageGenerationModel

Generates images from text prompt.

Examples::

model = ImageGenerationModel.from_pretrained("imagegeneration@002")
response = model.generate_images(
    prompt="Astronaut riding a horse",
    # Optional:
    number_of_images=1,
    seed=0,
)
response[0].show()
response[0].save("image1.png")

ImageGenerationResponse

Image generation response.

ImageQnAModel

Answers questions about an image.

Examples::

model = ImageQnAModel.from_pretrained("imagetext@001")
image = Image.load_from_file("image.png")
answers = model.ask_question(
    image=image,
    question="What color is the car in this image?",
    # Optional:
    number_of_results=1,
)

ImageTextModel

Generates text from images.

Examples::

model = ImageTextModel.from_pretrained("imagetext@001")
image = Image.load_from_file("image.png")

captions = model.get_captions(
    image=image,
    # Optional:
    number_of_results=1,
    language="en",
)

answers = model.ask_question(
    image=image,
    question="What color is the car in this image?",
    # Optional:
    number_of_results=1,
)

MultiModalEmbeddingModel

Generates embedding vectors from images and videos.

Examples::

model = MultiModalEmbeddingModel.from_pretrained("multimodalembedding@001")
image = Image.load_from_file("image.png")
video = Video.load_from_file("video.mp4")

embeddings = model.get_embeddings(
    image=image,
    video=video,
    contextual_text="Hello world",
)
image_embedding = embeddings.image_embedding
video_embeddings = embeddings.video_embeddings
text_embedding = embeddings.text_embedding

MultiModalEmbeddingResponse

The multimodal embedding response.

Video

Video.

VideoEmbedding

Embeddings generated from video with offset times.

VideoSegmentConfig

The specific video segments (in seconds) the embeddings are generated for.

Modules

_language_models

Classes for working with language models.

generative_models

Classes for working with the Gemini models.

language_models

Classes for working with language models.

sft

Classes for supervised tuning.

vision_models

Classes for working with vision models.