评估 AutoML 模型

Vertex AI 提供了模型评估指标，可帮助您确定模型的性能，例如精确率和召回率指标。Vertex AI 使用测试集来计算评估指标。

如何使用模型评估指标

模型评估指标提供模型在测试集上的表现的量化测量结果。如何解读和使用这些指标取决于您的业务需求和模型训练所要解决的问题。例如，假正例的容忍度可能低于假负例，反之亦然。这些问题会影响您重点关注的指标。

如需详细了解如何迭代模型以改善其性能，请参阅迭代模型。

示例

以下场景介绍了如何使用评估指标来确定 AutoML 模型的性能。

图像中的隐私权

假设您要创建一个自动检测图像中的敏感信息的系统并对其进行模糊处理。在这种情况下，假正例是指不需要模糊处理但仍然进行模糊处理等情况，例如日历上的数字。此操作不太方便，但并不会带来负面影响。

在此用例中，假负例是指未能进行模糊处理的内容，例如信用卡号，可能导致身份盗用。

在这种情况下，您可以专注于召回率。召回率测量是指，预测出现的所有预测遗漏了多少数据。较高的召回率会增加假正例的数量（降低精确率），但减少假负例的数量。换句话说，该模型可能还包含更多相关对象，但相关性也更不相关。

垃圾邮件过滤

假设您要创建一个自动过滤垃圾邮件信息的系统。 在此用例中，垃圾邮件未被捕获，从而出现在收件箱中，这就是一个假负例。对于用户而言，这会略有不便。

如果一封电子邮件被错误地标记为垃圾邮件而清除出收件箱，即为本用例中的假正例。如果电子邮件至关重要，则用户可能会受到严重影响。

在这种情况下，您需要关注精确率，这反映了预测的总预测数量。高精确率的模型可能只会为最相关的样本添加类别，这对于类别在训练数据中很常见的情况非常有用。

Vertex AI 返回的评估指标

Vertex AI 会返回几个不同的评估指标，例如精确率、召回率和置信度阈值。Vertex AI 返回的指标取决于您的模型目标。例如，与图像对象检测模型相比，Vertex AI 为图像分类模型提供了不同的评估指标。

架构文件可从 Cloud Storage 位置下载，它决定了 Vertex AI 为每个目标提供的评估指标。以下标签提供了架构文件的链接，并描述了每个模型目标的评估指标。

图像

分类

您可以从以下 Cloud Storage 位置查看和下载架构文件：
gs://google-cloud-aiplatform/schema/modelevaluation/

• AuPRC：精确率/召回率 (PR) 曲线下的面积，亦称为平均精度。此值的范围在 0 到 1 之间，值越大表示模型质量越高。
• 对数损失：模型预测与目标值之间的交叉熵。此值的范围在零到无穷大之间，值越小表示模型质量越高。
• 置信度阈值：用于确定要返回的预测的置信度分数。模型会返回不低于此值的预测。如果提高置信度阈值，精确率会相应地提高，但召回率会降低。Vertex AI 以不同的阈值返回置信度指标，以显示阈值如何影响精确率和召回率。
• 召回率：模型正确预测此类别的预测所占的比例。也称为真正例率。
• 精确率：模型生成的正确分类预测所占的比例。
• 混淆矩阵：混淆矩阵显示模型正确预测结果的频率。如果预测结果不正确，矩阵会改为显示模型预测的内容。混淆矩阵可帮助您了解模型“混淆”两个结果的出现位置。

对象检测

您可以从以下 Cloud Storage 位置查看和下载架构文件：
gs://google-cloud-aiplatform/schema/modelevaluation/

• IoU 阈值：用于确定返回哪些预测的交并比阈值。模型会返回不低于此值的预测。阈值越高，预测边界框值必须越接近实际的边界框值。
• 平均精确率均值：也称为平均精确率。此值的范围在 0 到 1 之间，值越大表示模型质量越高。
• 置信度阈值：用于确定要返回的预测的置信度分数。模型会返回不低于此值的预测。如果提高置信度阈值，精确率会相应地提高，但召回率会降低。Vertex AI 以不同的阈值返回置信度指标，以显示阈值如何影响精确率和召回率。
• 召回率：模型正确预测此类别的预测所占的比例。也称为真正例率。
• 精确率：模型生成的正确分类预测所占的比例。
• F1 得分：精确率和召回率的调和平均数。F1 是一个很实用的指标，当您希望在精确率和召回率之间取得平衡，而类别分布又不均匀时，该指标会非常有帮助。
• 边界框平均精度均值：边界框评估的单一指标：所有 boundingBoxMetrics 的平均值 meanAveragePrecision。

表格

分类

您可以从以下 Cloud Storage 位置查看和下载架构文件：
gs://google-cloud-aiplatform/schema/modelevaluation/

• AuPRC：精确率/召回率 (PR) 曲线下的面积，亦称为平均精度。此值的范围在 0 到 1 之间，值越大表示模型质量越高。
• AuROC：接收者操作特征曲线下的面积。此值的范围在 0 到 1 之间，值越大表示模型质量越高。
• 对数损失：模型预测与目标值之间的交叉熵。此值的范围在零到无穷大之间，值越小表示模型质量越高。
• 置信度阈值：用于确定要返回的预测的置信度分数。模型会返回不低于此值的预测。如果提高置信度阈值，精确率会相应地提高，但召回率会降低。Vertex AI 以不同的阈值返回置信度指标，以显示阈值如何影响精确率和召回率。
• 召回率：模型正确预测此类别的预测所占的比例。也称为真正例率。
• 召回率为 1：仅考虑具有最高预测分数且不低于每个样本的置信度阈值的标签时的召回率（真正例率）。
• 精确率：模型生成的正确分类预测所占的比例。
• 精确率为 1：仅考虑具有最高预测分数且不低于每个样本的置信度阈值的标签时的精确率。
• F1 得分：精确率和召回率的调和平均数。F1 是一个很实用的指标，当您希望在精确率和召回率之间取得平衡，而类别分布又不均匀时，该指标会非常有帮助。
• F1 得分为 1：召回率为 1 且精确率为 1 时的调和平均数。
• 真负例计数：模型正确预测负类别的次数。
• 真正例计数：模型正确预测正类别的次数。
• 假负例计数：模型错误预测负类别的次数。
• 假正例计数：模型错误预测正类别的次数。
• 假正例率：不正确预测结果占所有预测结果的比例。
• 假正例率为 1：仅考虑具有最高预测分数且不低于每个样本的置信度阈值的标签时的假正例率。
• 混淆矩阵：混淆矩阵显示模型正确预测结果的频率。如果预测结果不正确，矩阵会改为显示模型预测的内容。混淆矩阵可帮助您了解模型“混淆”两个结果的出现位置。
• 模型特征归因：Vertex AI 显示每个特征对模型的影响程度。这些值是以每个特征的百分比形式提供的：百分比越高，特征对模型训练的影响就越大。应查看此信息，以确保所有最重要的特征对您的数据和业务问题都有所帮助。

预测

您可以从以下 Cloud Storage 位置查看和下载架构文件：
gs://google-cloud-aiplatform/schema/modelevaluation/

• MAE：平均绝对误差 (MAE) 指的是目标值与预测值之间的平均绝对差。此指标的范围在零到无穷大之间；值越小表示模型质量越高。
• RMSE：均方根误差是目标值与预测值之间的平均平方差的平方根。RMSE 对离群值比 MAE 更敏感，因此如果您担心大的误差，那么 RMSE 可能是一个更有用的评估指标。 与 MAE 类似，值越小表示模型质量越高（0 表示完美的预测模型）。
• RMSLE：均方根对数误差指标与 RMSE 类似，不同的是它使用预测值和实际值加 1 的自然对数。RMSLE 对预测不足的罚分比过度预测更重。如果您不希望对大预测值误差的罚分比对小预测值的更重，则此指标也非常适合您的需求。此指标的范围在零到无穷大之间；值越小表示模型质量越高。只有当所有标签值和预测值均为非负值时，才会返回 RMSLE 评估指标。
• r^2：r 平方 (r^2) 是标签值与预测值之间的皮尔逊相关系数的平方，该指标的范围介于 0 和 1 之间；值越高表示模型质量越高。
• MAPE：平均绝对百分比误差 (MAPE) 指的是标签与预测值之间的平均绝对百分比差。该指标的范围介于 0 到无穷大之间；值越低表示模型质量越高。
  如果目标列包含任何为 0 的值，则 MAPE 不会显示。在这种情况下，MAPE 未经定义。
• WAPE：加权绝对百分比误差 (WAPE) 指的是模型预测的值与观察到的值之间的总体差值。与 RMSE 相比，WAPE 更侧重于总体差异而非单个差异，而这些差异在很大程度上受较低或间歇性值的影响。值越小表示模型质量越高。
• RMSPE：均方根误差百分比 (RMPSE) 将 RMSE 显示为实际值的百分比，而不是绝对值。值越小表示模型质量越高。
• 分位数：百分比分位数，表示观察到的值将低于预测值的概率。例如，在 0.2 分位数处，观察到的值预计会比预测值低 20%。如果您为优化目标指定 minimize-quantile-loss，则 Vertex AI 会提供此指标。
• 观察到的分位数：显示小于给定分位数的预测值 true 值的百分比。如果您为优化目标指定 minimize-quantile-loss，则 Vertex AI 会提供此指标。
• 弹力 Pinball 损失：特定分位数的弹力 Pinball 损失。分值越低，表示给定分位数质量模型越高。如果您为优化目标指定 minimize-quantile-loss，则 Vertex AI 会提供此指标。

回归

您可以从以下 Cloud Storage 位置查看和下载架构文件：
gs://google-cloud-aiplatform/schema/modelevaluation/

• MAE：平均绝对误差 (MAE) 指的是目标值与预测值之间的平均绝对差。此指标的范围在零到无穷大之间；值越小表示模型质量越高。
• RMSE：均方根误差是目标值与预测值之间的平均平方差的平方根。RMSE 对离群值比 MAE 更敏感，因此如果您担心大的误差，那么 RMSE 可能是一个更有用的评估指标。 与 MAE 类似，值越小表示模型质量越高（0 表示完美的预测模型）。
• RMSLE：均方根对数误差指标与 RMSE 类似，不同的是它使用预测值和实际值加 1 的自然对数。RMSLE 对预测不足的罚分比过度预测更重。如果您不希望对大预测值误差的罚分比对小预测值的更重，则此指标也非常适合您的需求。此指标的范围在零到无穷大之间；值越小表示模型质量越高。只有当所有标签值和预测值均为非负值时，才会返回 RMSLE 评估指标。
• r^2：r 平方 (r^2) 是标签值与预测值之间的皮尔逊相关系数的平方，该指标的范围介于 0 和 1 之间；值越高表示模型质量越高。
• MAPE：平均绝对百分比误差 (MAPE) 指的是标签与预测值之间的平均绝对百分比差。该指标的范围介于 0 到无穷大之间；值越低表示模型质量越高。
  如果目标列包含任何为 0 的值，则 MAPE 不会显示。在这种情况下，MAPE 未经定义。
• 模型特征归因：Vertex AI 显示每个特征对模型的影响程度。这些值是以每个特征的百分比形式提供的：百分比越高，特征对模型训练的影响就越大。应查看此信息，以确保所有最重要的特征对您的数据和业务问题都有所帮助。

文本

分类

您可以从以下 Cloud Storage 位置查看和下载架构文件：
gs://google-cloud-aiplatform/schema/modelevaluation/

• AuPRC：精确率/召回率 (PR) 曲线下的面积，亦称为平均精度。此值的范围在 0 到 1 之间，值越大表示模型质量越高。
• 对数损失：模型预测与目标值之间的交叉熵。此值的范围在零到无穷大之间，值越小表示模型质量越高。
• 置信度阈值：用于确定要返回的预测的置信度分数。模型会返回不低于此值的预测。如果提高置信度阈值，精确率会相应地提高，但召回率会降低。Vertex AI 以不同的阈值返回置信度指标，以显示阈值如何影响精确率和召回率。
• 召回率：模型正确预测此类别的预测所占的比例。也称为真正例率。
• 召回率为 1：仅考虑具有最高预测分数且不低于每个样本的置信度阈值的标签时的召回率（真正例率）。
• 精确率：模型生成的正确分类预测所占的比例。
• 精确率为 1：仅考虑具有最高预测分数且不低于每个样本的置信度阈值的标签时的精确率。
• F1 得分：精确率和召回率的调和平均数。F1 是一个很实用的指标，当您希望在精确率和召回率之间取得平衡，而类别分布又不均匀时，该指标会非常有帮助。
• F1 得分为 1：召回率为 1 且精确率为 1 时的调和平均数。
• 混淆矩阵：混淆矩阵显示模型正确预测结果的频率。如果预测结果不正确，矩阵会改为显示模型预测的内容。混淆矩阵可帮助您了解模型“混淆”两个结果的出现位置。

实体提取

您可以从以下 Cloud Storage 位置查看和下载架构文件：
gs://google-cloud-aiplatform/schema/modelevaluation/

• 置信度阈值：用于确定要返回的预测的置信度分数。模型会返回不低于此值的预测。如果提高置信度阈值，精确率会相应地提高，但召回率会降低。Vertex AI 以不同的阈值返回置信度指标，以显示阈值如何影响精确率和召回率。
• 召回率：模型正确预测此类别的预测所占的比例。也称为真正例率。
• 精确率：模型生成的正确分类预测所占的比例。
• F1 得分：精确率和召回率的调和平均数。F1 是一个很实用的指标，当您希望在精确率和召回率之间取得平衡，而类别分布又不均匀时，该指标会非常有帮助。
• 混淆矩阵：混淆矩阵显示模型正确预测结果的频率。如果预测结果不正确，矩阵会改为显示模型预测的内容。混淆矩阵可帮助您了解模型“混淆”两个结果的出现位置。

情感分析

您可以从以下 Cloud Storage 位置查看和下载架构文件：
gs://google-cloud-aiplatform/schema/modelevaluation/

• 召回率：模型正确预测此类别的预测所占的比例。也称为真正例率。
• 精确率：模型生成的正确分类预测所占的比例。
• F1 得分：精确率和召回率的调和平均数。F1 是一个很实用的指标，当您希望在精确率和召回率之间取得平衡，而类别分布又不均匀时，该指标会非常有帮助。
• MAE：平均绝对误差 (MAE) 指的是目标值与预测值之间的平均绝对差。此指标的范围在零到无穷大之间；值越小表示模型质量越高。
• MSE：均方误差 (MSE) 衡量模型预测值或估计值与观察值之间的差异。数值越低，表示模型越准确。
• 线性加权 kappa 和二次加权 kappa 衡量模型分配的情感值与人工标注者分配的情感值的接近程度。数值越高，表示模型越准确。
• 混淆矩阵：混淆矩阵显示模型正确预测结果的频率。如果预测结果不正确，矩阵会改为显示模型预测的内容。混淆矩阵可帮助您了解模型“混淆”两个结果的出现位置。

视频

动作识别

您可以从以下 Cloud Storage 位置查看和下载架构文件：
gs://google-cloud-aiplatform/schema/modelevaluation/

• AuPRC：精确率/召回率 (PR) 曲线下的面积，亦称为平均精度。此值的范围在 0 到 1 之间，值越大表示模型质量越高。
• 精确率窗口长度：预测的时间戳必须在此窗口长度内才视为真正例。精确率窗口长度的中心点是包含此特定长度的真实操作的时间戳。该值以从视频开始时间算起的秒数表示，并在末尾附加“s”。允许使用小数，精确到微秒。
• 平均精确率均值：也称为平均精确率。此值的范围在 0 到 1 之间，值越大表示模型质量越高。
• 置信度阈值：用于确定要返回的预测的置信度分数。模型会返回不低于此值的预测。如果提高置信度阈值，精确率会相应地提高，但召回率会降低。Vertex AI 以不同的阈值返回置信度指标，以显示阈值如何影响精确率和召回率。
• 召回率：模型正确预测此类别的预测所占的比例。也称为真正例率。
• 精确率：模型生成的正确分类预测所占的比例。
• F1 得分：精确率和召回率的调和平均数。F1 是一个很实用的指标，当您希望在精确率和召回率之间取得平衡，而类别分布又不均匀时，该指标会非常有帮助。

分类

您可以从以下 Cloud Storage 位置查看和下载架构文件：
gs://google-cloud-aiplatform/schema/modelevaluation/

• AuPRC：精确率/召回率 (PR) 曲线下的面积，亦称为平均精度。此值的范围在 0 到 1 之间，值越大表示模型质量越高。
• 置信度阈值：用于确定要返回的预测的置信度分数。模型会返回不低于此值的预测。如果提高置信度阈值，精确率会相应地提高，但召回率会降低。Vertex AI 以不同的阈值返回置信度指标，以显示阈值如何影响精确率和召回率。
• 召回率：模型正确预测此类别的预测所占的比例。也称为真正例率。
• 精确率：模型生成的正确分类预测所占的比例。
• F1 得分：精确率和召回率的调和平均数。F1 是一个很实用的指标，当您希望在精确率和召回率之间取得平衡，而类别分布又不均匀时，该指标会非常有帮助。
• 混淆矩阵：混淆矩阵显示模型正确预测结果的频率。如果预测结果不正确，矩阵会改为显示模型预测的内容。混淆矩阵可帮助您了解模型“混淆”两个结果的出现位置。

对象跟踪

您可以从以下 Cloud Storage 位置查看和下载架构文件：
gs://google-cloud-aiplatform/schema/modelevaluation/

• AuPRC：精确率/召回率 (PR) 曲线下的面积，亦称为平均精度。此值的范围在 0 到 1 之间，值越大表示模型质量越高。
• IoU 阈值：用于确定返回哪些预测的交并比阈值。模型会返回不低于此值的预测。阈值越高，预测边界框值必须越接近实际的边界框值。
• 平均精确率均值：也称为平均精确率。此值的范围在 0 到 1 之间，值越大表示模型质量越高。
• 置信度阈值：用于确定要返回的预测的置信度分数。模型会返回不低于此值的预测。如果提高置信度阈值，精确率会相应地提高，但召回率会降低。Vertex AI 以不同的阈值返回置信度指标，以显示阈值如何影响精确率和召回率。
• 召回率：模型正确预测此类别的预测所占的比例。也称为真正例率。
• 精确率：模型生成的正确分类预测所占的比例。
• F1 得分：精确率和召回率的调和平均数。F1 是一个很实用的指标，当您希望在精确率和召回率之间取得平衡，而类别分布又不均匀时，该指标会非常有帮助。
• 边界框平均精度均值：边界框评估的单一指标：所有 boundingBoxMetrics 的平均值 meanAveragePrecision。

获取评估指标

您可以获取模型的一组总体评估指标，以及针对特定类或标签的评估指标。特定类或标签的评估指标也称为评估切片。以下内容介绍如何使用 Google Cloud Console 或 API 获取总体评估指标和评估切片。

控制台

1. 在 Google Cloud 控制台的 Vertex AI 部分中，前往模型页面。

   转到“模型”页面

2. 在区域下拉列表中，选择您的模型所在的区域。

3. 在模型列表中，点击您的模型以打开模型的评估标签页。

   在评估标签页中，您可以查看模型的总体评估指标，例如平均精确率和召回率。

   如果模型目标具有评估切片，则控制台会显示标签列表。您可以点击标签来查看该标签的评估指标，如以下示例所示：

   

API

对于每种数据类型和目标，获取评估指标的 API 请求是相同的，但输出有所不同。以下示例显示了相同的请求和不同的响应。

获取聚合模型评估指标

总体模型评估指标提供有关整个模型的信息。如需查看特定切片的相关信息，请列出模型评估切片。

如需查看总体模型评估指标，请使用 projects.locations.models.evaluations.get 方法。

图像

在下面选择您的目标对应的标签页：

分类

Vertex AI 会返回一组置信度指标。每个元素显示不同 confidenceThreshold 值的评估指标（从 0 开始，一直到 1）。通过查看不同的阈值，您可以看到阈值如何影响其他指标，例如精确率和召回率。

选择与您的语言或环境对应的标签页：

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：存储模型的区域。
• PROJECT：您的项目 ID。
• MODEL_ID：模型资源的 ID。
• PROJECT_NUMBER：您项目的编号。
• EVALUATION_ID：模型评估的 ID（显示在响应中）。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 0.9946683,
        "logLoss": 0.043421753,
        "confidenceMetrics": [
          {
            "recall": 1,
            "precision": 0.2
          },
          {
            "confidenceThreshold": 0.05,
            "recall": 0.98637605,
            "precision": 0.8960396
          },
          {
            "confidenceThreshold": 0.1,
            "recall": 0.98637605,
            "precision": 0.92346936
          },
          {
            "confidenceThreshold": 0.15,
            "recall": 0.9782016,
            "precision": 0.93979055
          },
          ...
        ],
        "confusionMatrix": {
          "annotationSpecs": [
            {
              "id": "216348703974227968",
              "displayName": "tulips"
            },
            {
              "id": "1369270208581074944",
              "displayName": "dandelion"
            },
            {
              "id": "2522191713187921920",
              "displayName": "sunflowers"
            },
            {
              "id": "4828034722401615872",
              "displayName": "daisy"
            },
            {
              "id": "7133877731615309824",
              "displayName": "roses"
            }
          ],
          "rows": [
            [
              78,
              0,
              0,
              0,
              2
            ],
            [
              0,
              89,
              0,
              1,
              0
            ],
            [
              1,
              0,
              67,
              1,
              1
            ],
            [
              0,
              1,
              0,
              61,
              1
            ],
            [
              6,
              0,
              0,
              0,
              58
            ]
          ]
        }
      },
      "createTime": "2020-10-08T23:35:54.559702Z",
      "sliceDimensions": [
        "annotationSpec"
      ]
    }
  ]
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationImageClassificationSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    getModelEvaluationImageClassificationSample(project, modelId, evaluationId);
  }

  static void getModelEvaluationImageClassificationSample(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Image Classification Response");
      System.out.format("Model Name: %s\n", modelEvaluation.getName());
      System.out.format("Metrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("Metrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("Create Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("Slice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationImageClassification() {
  // Configure the name resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Create get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation image classification response');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);
  console.log(`\tSlice dimensions : ${response.sliceDimensions}`);

  const modelExplanation = response.modelExplanation;
  if (modelExplanation === null) {
    console.log(`\tModel explanation: ${JSON.stringify(modelExplanation)}`);
  } else {
    const meanAttributions = modelExplanation.meanAttributions;
    for (const meanAttribution of meanAttributions) {
      console.log('\t\tMean attribution');
      console.log(
        `\t\t\tBaseline output value : \
           ${meanAttribution.baselineOutputValue}`
      );
      console.log(
        `\t\t\tInstance output value : \
           ${meanAttribution.instanceOutputValue}`
      );
      console.log(
        `\t\t\tFeature attributions : \
           ${JSON.stringify(meanAttribution.featureAttributions)}`
      );
      console.log(`\t\t\tOutput index : ${meanAttribution.outputIndex}`);
      console.log(
        `\t\t\tOutput display name : \
           ${meanAttribution.outputDisplayName}`
      );
      console.log(
        `\t\t\tApproximation error : \
           ${meanAttribution.approximationError}`
      );
    }
  }
}
getModelEvaluationImageClassification();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def get_model_evaluation_image_classification_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

对象检测

对于边界框指标，Vertex AI 会返回一组采用不同 IoU 阈值（介于 0 和 1 之间）和置信度阈值（介于 0 和 1 之间）的指标值。例如，您可以采用 IoU 阈值 0.85 和置信度阈值 0.8228 来缩小评估指标范围。通过查看这些不同的阈值，您可以看到它们如何影响其他指标，例如精确率和召回率。

选择与您的语言或环境对应的标签页：

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：存储模型的区域。
• PROJECT：您的项目 ID。
• MODEL_ID：模型资源的 ID。
• PROJECT_NUMBER：您项目的编号。
• EVALUATION_ID：模型评估的 ID（显示在响应中）。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/image_object_detection_metrics_1.0.0.yaml",
      "metrics": {
        "evaluatedBoundingBoxCount": 190,
        "boundingBoxMetrics": [
          {
            "iouThreshold": 0.85,
            "meanAveragePrecision": 0.0818413,
            "confidenceMetrics": [
              {
                "confidenceThreshold": 7.7341465e-06,
                "recall": 0.22105263,
                "precision": 0.045652173,
                "f1Score": 0.075675674
              },
              {
                "confidenceThreshold": 0.017749911,
                "recall": 0.21578947,
                "precision": 0.078998074,
                "f1Score": 0.11565586
              },
              {
                "confidenceThreshold": 0.022688486,
                "recall": 0.21052632,
                "precision": 0.08583691,
                "f1Score": 0.121951215
              },
              ...
        ],
        "boundingBoxMeanAveragePrecision": 0.3128878
      },
      "createTime": "2020-10-09T23:35:43.325822Z",
      "sliceDimensions": [
        "annotationSpec"
      ]
    }
  ]
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationImageObjectDetectionSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    getModelEvaluationImageObjectDetectionSample(project, modelId, evaluationId);
  }

  static void getModelEvaluationImageObjectDetectionSample(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Image Object Detection Response");
      System.out.format("\tName: %s\n", modelEvaluation.getName());
      System.out.format("\tMetrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("\tMetrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("\tCreate Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("\tSlice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationImageObjectDetection() {
  // Configure the name resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Create get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation image object detection response');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);
  console.log(`\tSlice dimensions : ${response.sliceDimensions}`);

  const modelExplanation = response.modelExplanation;
  console.log('\tModel explanation');
  if (modelExplanation === null) {
    console.log('\t\t{}');
  } else {
    const meanAttributions = modelExplanation.meanAttributions;
    if (meanAttributions === null) {
      console.log('\t\t\t []');
    } else {
      for (const meanAttribution of meanAttributions) {
        console.log('\t\tMean attribution');
        console.log(
          `\t\t\tBaseline output value : \
            ${meanAttribution.baselineOutputValue}`
        );
        console.log(
          `\t\t\tInstance output value : \
            ${meanAttribution.instanceOutputValue}`
        );
        console.log(
          `\t\t\tFeature attributions : \
            ${meanAttribution.featureAttributions}`
        );
        console.log(`\t\t\tOutput index : ${meanAttribution.outputIndex}`);
        console.log(
          `\t\t\tOutput display name : \
            ${meanAttribution.outputDisplayName}`
        );
        console.log(
          `\t\t\tApproximation error : \
            ${meanAttribution.approximationError}`
        );
      }
    }
  }
}
getModelEvaluationImageObjectDetection();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def get_model_evaluation_image_object_detection_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

表格

在下面选择您的目标对应的标签页：

分类

Vertex AI 会返回一组置信度指标。每个元素显示不同 confidenceThreshold 值的评估指标（从 0 开始，一直到 1）。通过查看不同的阈值，您可以看到阈值如何影响其他指标，例如精确率和召回率。

选择与您的语言或环境对应的标签页：

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：存储模型的区域。
• PROJECT：您的项目 ID。
• MODEL_ID：模型资源的 ID。
• PROJECT_NUMBER：您项目的编号。
• EVALUATION_ID：模型评估的 ID（显示在响应中）。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 0.97762364,
        "auRoc": 0.97566897,
        "logLoss": 0.19153881,
        "confidenceMetrics": [
          {
            "recall": 1,
            "precision": 0.5,
            "falsePositiveRate": 1,
            "f1Score": 0.6666667,
            "recallAt1": 0.90911126,
            "precisionAt1": 0.90911126,
            "falsePositiveRateAt1": 0.09088874,
            "f1ScoreAt1": 0.90911126,
            "truePositiveCount": "4467",
            "falsePositiveCount": "4467"
          },
          {
            "confidenceThreshold": 0.003269856,
            "recall": 0.9997761,
            "precision": 0.56993365,
            "falsePositiveRate": 0.7544213,
            "f1Score": 0.7260018,
            "recallAt1": 0.90911126,
            "precisionAt1": 0.90911126,
            "falsePositiveRateAt1": 0.09088874,
            "f1ScoreAt1": 0.90911126,
            "truePositiveCount": "4466",
            "falsePositiveCount": "3370",
            "falseNegativeCount": "1",
            "trueNegativeCount": "1097"
          },
          {
            "confidenceThreshold": 0.1103351,
            "recall": 0.9899261,
            "precision": 0.79819494,
            "falsePositiveRate": 0.25027984,
            "f1Score": 0.8837814,
            "recallAt1": 0.90911126,
            "precisionAt1": 0.90911126,
            "falsePositiveRateAt1": 0.09088874,
            "f1ScoreAt1": 0.90911126,
            "truePositiveCount": "4422",
            "falsePositiveCount": "1118",
            "falseNegativeCount": "45",
            "trueNegativeCount": "3349"
          },
          ...
        ],
        "confusionMatrix": {
          "annotationSpecs": [
            {
              "displayName": "1"
            },
            {
              "displayName": "2"
            }
          ],
          "rows": [
            [
              3817,
              140
            ],
            [
              266,
              244
            ]
          ]
        }
      },
      "createTime": "2020-10-09T00:19:15.463930Z",
      "sliceDimensions": [
        "annotationSpec"
      ],
      "modelExplanation": {
        "meanAttributions": [
          {
            "featureAttributions": {
              "Age": 0.022972771897912025,
              "Job": 0.031542550772428513,
              "MaritalStatus": 0.015506803058087826,
              "Education": 0.019189134240150452,
              "Default": 0.00021766019926872104,
              "Balance": 0.031217793002724648,
              "Housing": 0.06786702573299408,
              "Loan": 0.0072592208161950111,
              "Contact": 0.083566240966320038,
              "Day": 0.074894927442073822,
              "Month": 0.19679982960224152,
              "Duration": 0.35500210523605347,
              "Campaign": 0.033425047993659973,
              "PDays": 0.013902961276471615,
              "Previous": 0.0061685866676270962,
              "POutcome": 0.040467333048582077
            }
          }
        ]
      }
    }
  ]
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationTabularClassificationSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    getModelEvaluationTabularClassification(project, modelId, evaluationId);
  }

  static void getModelEvaluationTabularClassification(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);
      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Tabular Classification Response");
      System.out.format("\tName: %s\n", modelEvaluation.getName());
      System.out.format("\tMetrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("\tMetrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("\tCreate Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("\tSlice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationTabularClassification() {
  // Configure the parent resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation tabular classification response');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);
  console.log(`\tSlice dimensions : ${response.sliceDimensions}`);

  const modelExplanation = response.modelExplanation;
  console.log('\tModel explanation');
  if (!modelExplanation) {
    console.log('\t\t{}');
  } else {
    const meanAttributions = modelExplanation.meanAttributions;
    if (!meanAttributions) {
      console.log('\t\t\t []');
    } else {
      for (const meanAttribution of meanAttributions) {
        console.log('\t\tMean attribution');
        console.log(
          `\t\t\tBaseline output value : \
            ${meanAttribution.baselineOutputValue}`
        );
        console.log(
          `\t\t\tInstance output value : \
            ${meanAttribution.instanceOutputValue}`
        );
        console.log(
          `\t\t\tFeature attributions : \
            ${JSON.stringify(meanAttribution.featureAttributions)}`
        );
        console.log(`\t\t\tOutput index : ${meanAttribution.outputIndex}`);
        console.log(
          `\t\t\tOutput display name : \
            ${meanAttribution.outputDisplayName}`
        );
        console.log(
          `\t\t\tApproximation error : \
            ${meanAttribution.approximationError}`
        );
      }
    }
  }
}
getModelEvaluationTabularClassification();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def get_model_evaluation_tabular_classification_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

预测

预览

本产品或功能须遵守服务专用条款的“通用服务条款”部分中的“非正式版产品条款”。 正式发布前产品和功能“按原样”提供，可能仅提供有限支持。 如需了解详情，请参阅发布阶段说明。

选择与您的语言或环境对应的标签页：

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：存储模型的区域。
• PROJECT：您的项目 ID。
• MODEL_ID：模型资源的 ID。模型训练成功完成后，MODEL_ID 会显示在训练流水线中。请参阅准备工作部分以获取 MODEL_ID。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/forecasting_metrics_1.0.0.yaml",
      "metrics": {
        "rootMeanSquaredError": 719.0045,
        "meanAbsoluteError": 487.0792,
        "meanAbsolutePercentageError": "Infinity",
        "rSquared": 0.8837288,
        "rootMeanSquaredLogError": "NaN",
        "quantileMetrics": [
          {
            "quantile": 0.2,
            "scaledPinballLoss": 157.34422,
            "observedQuantile": 0.15918367346938775
          },
          {
            "quantile": 0.5,
            "scaledPinballLoss": 243.5396,
            "observedQuantile": 0.45551020408163267
          },
          {
            "quantile": 0.8,
            "scaledPinballLoss": 175.39418,
            "observedQuantile": 0.81183673469387752
          }
        ]
      },
      "createTime": "2021-04-13T01:00:54.091953Z"
    }
  ]
}

回归

选择与您的语言或环境对应的标签页：

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：存储模型的区域。
• PROJECT：您的项目 ID。
• MODEL_ID：模型资源的 ID。
• PROJECT_NUMBER：您项目的编号。
• EVALUATION_ID：模型评估的 ID（显示在响应中）。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/regression_metrics_1.0.0.yaml",
      "metrics": {
        "rootMeanSquaredError": 2553.6309,
        "meanAbsoluteError": 1373.3932,
        "meanAbsolutePercentageError": "Infinity",
        "rSquared": 0.060764354,
        "rootMeanSquaredLogError": "NaN"
      },
      "createTime": "2020-10-09T01:20:37.045482Z",
      "modelExplanation": {
        "meanAttributions": [
          {
            "featureAttributions": {
              "Age": 0.22535169124603271,
              "Job": 0.049311652779579163,
              "MaritalStatus": 0.033439181745052338,
              "Education": 0.10934026539325714,
              "Default": 0.021301545202732086,
              "Housing": 0.0631907731294632,
              "Loan": 0.055760543793439865,
              "Contact": 0.010930608958005905,
              "Day": 0.14066702127456665,
              "Month": 0.17570944130420685,
              "Duration": 0.054339192807674408,
              "Campaign": 0.015468073077499866,
              "PDays": 0.020416950806975365,
              "Previous": 0.0037290120963007212,
              "POutcome": 0.0040646209381520748,
              "Deposit": 0.016979435458779335
            }
          }
        ]
      }
    }
  ]
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationTabularRegressionSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    getModelEvaluationTabularRegression(project, modelId, evaluationId);
  }

  static void getModelEvaluationTabularRegression(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);
      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Tabular Regression Response");
      System.out.format("\tName: %s\n", modelEvaluation.getName());
      System.out.format("\tMetrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("\tMetrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("\tCreate Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("\tSlice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationTabularRegression() {
  // Configure the parent resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation tabular regression response');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);
  console.log(`\tSlice dimensions : ${response.sliceDimensions}`);

  const modelExplanation = response.modelExplanation;
  console.log('\tModel explanation');
  if (!modelExplanation) {
    console.log('\t\t{}');
  } else {
    const meanAttributions = modelExplanation.meanAttributions;
    if (!meanAttributions) {
      console.log('\t\t\t []');
    } else {
      for (const meanAttribution of meanAttributions) {
        console.log('\t\tMean attribution');
        console.log(
          `\t\t\tBaseline output value : \
            ${meanAttribution.baselineOutputValue}`
        );
        console.log(
          `\t\t\tInstance output value : \
            ${meanAttribution.instanceOutputValue}`
        );
        console.log(
          `\t\t\tFeature attributions : \
            ${JSON.stringify(meanAttribution.featureAttributions)}`
        );
        console.log(`\t\t\tOutput index : ${meanAttribution.outputIndex}`);
        console.log(
          `\t\t\tOutput display name : \
            ${meanAttribution.outputDisplayName}`
        );
        console.log(
          `\t\t\tApproximation error : \
            ${meanAttribution.approximationError}`
        );
      }
    }
  }
}
getModelEvaluationTabularRegression();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def get_model_evaluation_tabular_regression_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

文本

在下面选择您的目标对应的标签页：

分类

Vertex AI 会返回一组置信度指标。每个元素显示不同 confidenceThreshold 值的评估指标（从 0 开始，一直到 1）。通过查看不同的阈值，您可以看到阈值如何影响其他指标，例如精确率和召回率。

选择与您的语言或环境对应的标签页：

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：存储模型的区域。
• PROJECT：您的项目 ID。
• MODEL_ID：模型资源的 ID。
• PROJECT_NUMBER：您项目的编号。
• EVALUATION_ID：模型评估的 ID（显示在响应中）。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 0.9569226,
        "logLoss": 0.08399241,
        "confidenceMetrics": [
          {
            "recall": 1,
            "precision": 0.14285715,
            "f1Score": 0.25,
            "recallAt1": 0.88270044,
            "precisionAt1": 0.88270044,
            "f1ScoreAt1": 0.88270044
          },
          {
            "confidenceThreshold": 0.05,
            "recall": 0.9772152,
            "precision": 0.6852071,
            "f1Score": 0.80556524,
            "recallAt1": 0.88270044,
            "precisionAt1": 0.88270044,
            "f1ScoreAt1": 0.88270044
          },
          {
            "confidenceThreshold": 0.1,
            "recall": 0.9662447,
            "precision": 0.75032765,
            "f1Score": 0.8447067,
            "recallAt1": 0.88270044,
            "precisionAt1": 0.88270044,
            "f1ScoreAt1": 0.88270044
          },
          ...
        ],
        "confusionMatrix": {
          "annotationSpecs": [
            {
              "id": "1349461407095062528",
              "displayName": "affection"
            },
            {
              "id": "2502382911701909504",
              "displayName": "nature"
            },
            {
              "id": "3655304416308756480",
              "displayName": "enjoy_the_moment"
            },
            {
              "id": "4808225920915603456",
              "displayName": "achievement"
            },
            {
              "id": "5961147425522450432",
              "displayName": "exercise"
            },
            {
              "id": "7114068930129297408",
              "displayName": "bonding"
            },
            {
              "id": "8266990434736144384",
              "displayName": "leisure"
            }
          ],
          "rows": [
            [
              404,
              0,
              6,
              12,
              0,
              1,
              0
            ],
            [
              1,
              20,
              0,
              4,
              0,
              1,
              0
            ],
            [
              5,
              0,
              104,
              23,
              2,
              1,
              5
            ],
            [
              18,
              1,
              14,
              334,
              2,
              2,
              6
            ],
            [
              0,
              0,
              0,
              1,
              19,
              0,
              0
            ],
            [
              8,
              0,
              0,
              5,
              0,
              117,
              0
            ],
            [
              2,
              0,
              10,
              8,
              0,
              1,
              48
            ]
          ]
        }
      },
      "createTime": "2020-08-29T03:05:14.099479Z",
      "sliceDimensions": [
        "annotationSpec"
      ]
    }
  ]
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationTextClassificationSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";

    getModelEvaluationTextClassificationSample(project, modelId, evaluationId);
  }

  static void getModelEvaluationTextClassificationSample(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";

      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);
      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Text Classification Response");
      System.out.format("\tModel Name: %s\n", modelEvaluation.getName());
      System.out.format("\tMetrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("\tMetrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("\tCreate Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("\tSlice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationTextClassification() {
  // Configure the resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation text classification response :');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);

  const modelExplanation = response.modelExplanation;
  console.log('\tModel explanation');
  if (modelExplanation === null) {
    console.log('\t\t{}');
  } else {
    const meanAttributions = modelExplanation.meanAttributions;
    if (meanAttributions === null) {
      console.log('\t\t\t []');
    } else {
      for (const meanAttribution of meanAttributions) {
        console.log('\t\tMean attribution');
        console.log(
          `\t\t\tBaseline output value : \
            ${meanAttribution.baselineOutputValue}`
        );
        console.log(
          `\t\t\tInstance output value : \
            ${meanAttribution.instanceOutputValue}`
        );
        console.log(
          `\t\t\tFeature attributions : \
            ${JSON.stringify(meanAttribution.featureAttributions)}`
        );
        console.log(`\t\t\tOutput index : ${meanAttribution.outputIndex}`);
        console.log(
          `\t\t\tOutput display name : \
            ${meanAttribution.outputDisplayName}`
        );
        console.log(
          `\t\t\tApproximation error : \
            ${meanAttribution.approximationError}`
        );
      }
    }
  }
}
getModelEvaluationTextClassification();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def get_model_evaluation_text_classification_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

实体提取

Vertex AI 会返回一组置信度指标。每个元素显示不同 confidenceThreshold 值的评估指标（从 0 开始，一直到 1）。通过查看不同的阈值，您可以看到阈值如何影响其他指标，例如精确率和召回率。

选择与您的语言或环境对应的标签页：

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：存储模型的区域。
• PROJECT：您的项目 ID。
• MODEL_ID：模型资源的 ID。
• PROJECT_NUMBER：您项目的编号。
• EVALUATION_ID：模型评估的 ID（显示在响应中）。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/text_extraction_metrics_1.0.0.yaml",
      "metrics": {
        "confidenceMetrics": [
          {
            "confidenceThreshold": 0.04,
            "recall": 0.78520286,
            "precision": 0.75632185,
            "f1Score": 0.7704918
          },
          {
            "confidenceThreshold": 0.96,
            "recall": 0.7422434,
            "precision": 0.816273,
            "f1Score": 0.7775
          },
          {
            "confidenceThreshold": 0.15,
            "recall": 0.78520286,
            "precision": 0.75632185,
            "f1Score": 0.7704918
          },
          ...
        ],
        "confusionMatrix": {
          "annotationSpecs": [
            {
              "displayName": "NULL"
            },
            {
              "id": "1022387484157280256",
              "displayName": "DiseaseClass"
            },
            {
              "id": "3328230493370974208",
              "displayName": "Modifier"
            },
            {
              "id": "5634073502584668160",
              "displayName": "CompositeMention"
            },
            {
              "id": "8516377264101785600",
              "displayName": "SpecificDisease"
            }
          ],
          "rows": [
            [
              0,
              24,
              24,
              1,
              35
            ],
            [
              10,
              32,
              2,
              0,
              11
            ],
            [
              10,
              0,
              89,
              0,
              1
            ],
            [
              2,
              0,
              0,
              7,
              0
            ],
            [
              46,
              6,
              2,
              0,
              201
            ]
          ]
        }
      },
      "createTime": "2020-08-21T03:10:06.645369Z",
      "sliceDimensions": [
        "annotationSpec"
      ]
    }
  ]
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationTextEntityExtractionSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";

    getModelEvaluationTextEntityExtractionSample(project, modelId, evaluationId);
  }

  static void getModelEvaluationTextEntityExtractionSample(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";

      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);
      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Text Entity Extraction Response");
      System.out.format("\tModel Name: %s\n", modelEvaluation.getName());
      System.out.format("\tMetrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("\tMetrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("\tCreate Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("\tSlice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationTextEntityExtraction() {
  // Configure the resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation text entity extraction response :');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);

  const modelExplanation = response.modelExplanation;
  console.log('\tModel explanation');
  if (modelExplanation === null) {
    console.log('\t\t{}');
  } else {
    const meanAttributions = modelExplanation.meanAttributions;
    if (meanAttributions === null) {
      console.log('\t\t\t []');
    } else {
      for (const meanAttribution of meanAttributions) {
        console.log('\t\tMean attribution');
        console.log(
          `\t\t\tBaseline output value : \
            ${meanAttribution.baselineOutputValue}`
        );
        console.log(
          `\t\t\tInstance output value : \
            ${meanAttribution.instanceOutputValue}`
        );
        console.log(
          `\t\t\tFeature attributions : \
            ${JSON.stringify(meanAttribution.featureAttributions)}`
        );
        console.log(`\t\t\tOutput index : ${meanAttribution.outputIndex}`);
        console.log(
          `\t\t\tOutput display name : \
            ${meanAttribution.outputDisplayName}`
        );
        console.log(
          `\t\t\tApproximation error : \
            ${meanAttribution.approximationError}`
        );
      }
    }
  }
}
getModelEvaluationTextEntityExtraction();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def get_model_evaluation_text_entity_extraction_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

情感分析

选择与您的语言或环境对应的标签页：

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：存储模型的区域。
• PROJECT：您的项目 ID。
• MODEL_ID：模型资源的 ID。
• PROJECT_NUMBER：您项目的编号。
• EVALUATION_ID：模型评估的 ID（显示在响应中）。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/text_sentiment_metrics_1.0.0.yaml",
      "metrics": {
        "precision": 0.2519685,
        "recall": 0.2519685,
        "f1Score": 0.2519685,
        "meanAbsoluteError": 1.7480315,
        "meanSquaredError": 5.2440944,
        "linearKappa": 0.1432999,
        "quadraticKappa": 0.20580281,
        "confusionMatrix": {
          "annotationSpecs": [
            {
              "id": "4003770437476548608",
              "displayName": "0"
            },
            {
              "id": "8615456455903936512",
              "displayName": "1"
            },
            {
              "id": "1121466675959431168",
              "displayName": "2"
            },
            {
              "id": "5733152694386819072",
              "displayName": "3"
            },
            {
              "id": "3427309685173125120",
              "displayName": "4"
            },
            {
              "id": "8038995703600513024",
              "displayName": "5"
            },
            {
              "id": "2274388180566278144",
              "displayName": "6"
            }
          ],
          "rows": [
            [
              0,
              2,
              0,
              0,
              0,
              1,
              0
            ],
            [
              0,
              19,
              3,
              0,
              0,
              8,
              0
            ],
            [
              0,
              13,
              3,
              0,
              0,
              7,
              0
            ],
            [
              0,
              8,
              1,
              0,
              0,
              9,
              0
            ],
            [
              0,
              9,
              0,
              0,
              0,
              11,
              0
            ],
            [
              0,
              15,
              1,
              0,
              0,
              10,
              0
            ],
            [
              0,
              0,
              1,
              0,
              0,
              6,
              0
            ]
          ]
        }
      },
      "createTime": "2020-08-20T04:07:41.036839Z",
      "sliceDimensions": [
        "annotationSpec"
      ]
    }
  ]
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationTextSentimentAnalysisSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";

    getModelEvaluationTextSentimentAnalysisSample(project, modelId, evaluationId);
  }

  static void getModelEvaluationTextSentimentAnalysisSample(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";

      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);
      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Text Sentiment Analysis Response");
      System.out.format("\tModel Name: %s\n", modelEvaluation.getName());
      System.out.format("\tMetrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("\tMetrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("\tCreate Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("\tSlice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationTextSentimentAnalysis() {
  // Configure the resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation text sentiment analysis response :');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);

  const modelExplanation = response.modelExplanation;
  console.log('\tModel explanation');
  if (modelExplanation === null) {
    console.log('\t\t{}');
  } else {
    const meanAttributions = modelExplanation.meanAttributions;
    if (meanAttributions === null) {
      console.log('\t\t\t []');
    } else {
      for (const meanAttribution of meanAttributions) {
        console.log('\t\tMean attribution');
        console.log(
          `\t\t\tBaseline output value : \
            ${meanAttribution.baselineOutputValue}`
        );
        console.log(
          `\t\t\tInstance output value : \
            ${meanAttribution.instanceOutputValue}`
        );
        console.log(
          `\t\t\tFeature attributions : \
            ${JSON.stringify(meanAttribution.featureAttributions)}`
        );
        console.log(`\t\t\tOutput index : ${meanAttribution.outputIndex}`);
        console.log(
          `\t\t\tOutput display name : \
            ${meanAttribution.outputDisplayName}`
        );
        console.log(
          `\t\t\tApproximation error : \
            ${meanAttribution.approximationError}`
        );
      }
    }
  }
}
getModelEvaluationTextSentimentAnalysis();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def get_model_evaluation_text_sentiment_analysis_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

视频

在下面选择您的目标对应的标签页：

动作识别

Vertex AI 会返回一系列视频操作识别指标。每个元素显示不同 precisionWindowLength 和 confidenceThreshold 值的评估指标。通过查看不同窗口长度和置信度阈值的评估指标，您可以看到它们如何影响其他指标，例如精确率和召回率。

选择与您的语言或环境对应的标签页：

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：存储模型的区域。
• PROJECT：您的项目 ID。
• MODEL_ID：模型资源的 ID。
• PROJECT_NUMBER：您项目的编号。
• EVALUATION_ID：模型评估的 ID（显示在响应中）。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/693884908213/locations/us-central1/models/2108872098167390208/evaluations/2501984888492654592",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/video_action_recognition_metrics_1.0.0.yaml",
      "metrics": {
        "evaluatedActionCount": 4,
        "videoActionMetrics": [
          {
            "precisionWindowLength": "1s",
            "meanAveragePrecision": 1,
            "confidenceMetrics": [
              {
                "confidenceThreshold": 0.8158053,
                "recall": 0.25,
                "precision": 1,
                "f1Score": 0.4
              },
              {
                "confidenceThreshold": 0.793864,
                "recall": 0.5,
                "precision": 1,
                "f1Score": 0.6666667
              },
              {
                "confidenceThreshold": 0.72200835,
                "recall": 0.75,
                "precision": 1,
                "f1Score": 0.85714287
              },
              ...
            ]
          },
          {
            "precisionWindowLength": "2s",
            "meanAveragePrecision": 1,
            "confidenceMetrics": [
              {
                "confidenceThreshold": 0.8158053,
                "recall": 0.25,
                "precision": 1,
                "f1Score": 0.4
              },
              {
                "confidenceThreshold": 0.793864,
                "recall": 0.5,
                "precision": 1,
                "f1Score": 0.6666667
              },
              ...
            ]
          }
        ]
      },
      "createTime": "2020-11-09T20:02:05.795036Z",
      "sliceDimensions": [
        "annotationSpec"
      ]
    }
  ]
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationVideoActionRecognitionSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "PROJECT";
    String modelId = "MODEL_ID";
    String evaluationId = "EVALUATION_ID";
    getModelEvaluationVideoActionRecognitionSample(project, modelId, evaluationId);
  }

  static void getModelEvaluationVideoActionRecognitionSample(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings settings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();
    String location = "us-central1";

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient client = ModelServiceClient.create(settings)) {
      ModelEvaluationName name = ModelEvaluationName.of(project, location, modelId, evaluationId);
      ModelEvaluation response = client.getModelEvaluation(name);
      System.out.format("response: %s\n", response);
    }
  }
}

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def get_model_evaluation_video_action_recognition_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

分类

Vertex AI 会返回一组置信度指标。每个元素显示不同 confidenceThreshold 值的评估指标（从 0 开始，一直到 1）。通过查看不同的阈值，您可以看到阈值如何影响其他指标，例如精确率和召回率。

选择与您的语言或环境对应的标签页：

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：存储模型的区域。
• PROJECT：您的项目 ID。
• MODEL_ID：模型资源的 ID。
• PROJECT_NUMBER：您项目的编号。
• EVALUATION_ID：模型评估的 ID（显示在响应中）。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 1,
        "confidenceMetrics": [
          {
            "confidenceThreshold": 0.103617236,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          {
            "confidenceThreshold": 0.104728825,
            "recall": 1,
            "precision": 0.2016129,
            "f1Score": 0.33557048
          },
          {
            "confidenceThreshold": 0.10573554,
            "recall": 1,
            "precision": 0.20283976,
            "f1Score": 0.3372681
          },
          ...
        ],
        "confusionMatrix": {
          "annotationSpecs": [
            {
              "id": "1004478638764064768",
              "displayName": "pullup"
            },
            {
              "id": "3310321647977758720",
              "displayName": "kick_ball"
            },
            {
              "id": "5616164657191452672",
              "displayName": "cartwheel"
            },
            {
              "id": "7922007666405146624",
              "displayName": "ride_horse"
            },
            {
              "id": "8498468418708570112",
              "displayName": "golf"
            }
          ],
          "rows": [
            [
              20,
              0,
              0,
              0,
              0
            ],
            [
              0,
              20,
              0,
              0,
              0
            ],
            [
              0,
              0,
              20,
              0,
              0
            ],
            [
              0,
              0,
              0,
              20,
              0
            ],
            [
              0,
              0,
              0,
              0,
              20
            ]
          ]
        }
      },
      "createTime": "2020-10-09T01:43:47.000284Z",
      "sliceDimensions": [
        "annotationSpec"
      ]
    }
  ]
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationVideoClassificationSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    getModelEvaluationVideoClassification(project, modelId, evaluationId);
  }

  static void getModelEvaluationVideoClassification(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Video Classification Response");
      System.out.format("Name: %s\n", modelEvaluation.getName());
      System.out.format("Metrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("Metrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("Create Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("Slice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationVideoClassification() {
  // Configure the parent resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Create get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation video classification response');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);
  console.log(`\tSlice dimensions : ${response.sliceDimensions}`);
}
getModelEvaluationVideoClassification();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def get_model_evaluation_video_classification_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

对象跟踪

对于边界框指标，Vertex AI 会返回一组采用不同 IoU 阈值（介于 0 和 1 之间）和置信度阈值（介于 0 和 1 之间）的指标值。例如，您可以采用 IoU 阈值 0.85 和置信度阈值 0.8228 来缩小评估指标范围。通过查看这些不同的阈值，您可以看到它们如何影响其他指标，例如精确率和召回率。

选择与您的语言或环境对应的标签页：

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：存储模型的区域。
• PROJECT：您的项目 ID。
• MODEL_ID：模型资源的 ID。
• PROJECT_NUMBER：您项目的编号。
• EVALUATION_ID：模型评估的 ID（显示在响应中）。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/video_object_tracking_metrics_1.0.0.yaml",
      "metrics": {
        "boundingBoxMetrics": [
          {
            "iouThreshold": 0.5,
            "meanAveragePrecision": 0.6128407,
            "confidenceMetrics": [
              {
                "recall": 1,
                "precision": 0.76,
                "f1Score": 0.8636364
              },
              {
                "confidenceThreshold": 0.29,
                "recall": 0.17293233,
                "precision": 0.35384616,
                "f1Score": 0.23232323
              },
              {
                "confidenceThreshold": 0.31350172,
                "recall": 0.15789473,
                "precision": 0.8076923,
                "f1Score": 0.26415095
              },
              ...
            ]
          }
        ],
        "boundingBoxMeanAveragePrecision": 0.6128407
      },
      "createTime": "2020-10-12T22:42:59.570300Z",
      "sliceDimensions": [
        "annotationSpec"
      ]
    }
  ]
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationVideoObjectTrackingSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    getModelEvaluationVideoObjectTracking(project, modelId, evaluationId);
  }

  static void getModelEvaluationVideoObjectTracking(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Video Object Tracking Response");
      System.out.format("Name: %s\n", modelEvaluation.getName());
      System.out.format("Metrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("Metrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("Create Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("Slice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationVideoObjectTracking() {
  // Configure the parent resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Create get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation video object tracking response');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);
  console.log(`\tSlice dimensions : ${response.sliceDimensions}`);
}
getModelEvaluationVideoObjectTracking();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def get_model_evaluation_video_object_tracking_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

列出所有评估切片

projects.locations.models.evaluations.slices.list 方法可列出您的模型的所有评估切片。您必须拥有模型的评估 ID，才能在查看总体评估指标时获得此 ID。

您可以使用模型评估切片来确定模型在特定标签上的表现。value 字段指示与指标对应的标签。

图像

在下面选择您的目标对应的标签页：

分类

Vertex AI 会返回一组置信度指标。每个元素显示不同 confidenceThreshold 值的评估指标（从 0 开始，一直到 1）。通过查看不同的阈值，您可以看到阈值如何影响其他指标，例如精确率和召回率。

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：模型所在的区域。例如 us-central1。
• PROJECT：您的项目 ID。
• MODEL_ID：您的模型的 ID。
• EVALUATION_ID：包含要列出评估切片的模型评估的 ID。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluationSlices": [
    {
      "name": "projects/123456789012/locations/us-central1/models/1232584574060974976/evaluations/1234057709820346880/slices/123455414369100579",
      "slice": {
        "dimension": "annotationSpec",
        "value": "daisy"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 0.99713016,
        "auRoc": 0.99942565,
        "logLoss": 0.025896452,
        "confidenceMetrics": [
          {
            "recall": 1,
            "precision": 0.17166212
          },
          {
            "confidenceThreshold": 0.05,
            "recall": 1,
            "precision": 0.9402985
          },
          {
            "confidenceThreshold": 0.1,
            "recall": 1,
            "precision": 0.95454544
          },
          ...
      },
      "createTime": "2020-10-08T23:35:54.770876Z"
    },
    {
      "name": "projects/123456789012/locations/us-central1/models/1232584574060974976/evaluations/1234057709820346880/slices/123455414369101230",
      "slice": {
        "dimension": "annotationSpec",
        "value": "sunflowers"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 0.99369764,
        "auRoc": 0.9978355,
        "logLoss": 0.049820196,
        "confidenceMetrics": [
          {
            "recall": 1,
            "precision": 0.1907357
          },
          {
            "confidenceThreshold": 0.05,
            "recall": 0.95714283,
            "precision": 0.9852941
          },
          {
            "confidenceThreshold": 0.1,
            "recall": 0.95714283,
            "precision": 0.9852941
          },
          ...
      },
      "createTime": "2020-10-08T23:35:54.770876Z"
    }
  ]
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class ListModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    listModelEvaluationSliceSample(project, modelId, evaluationId);
  }

  static void listModelEvaluationSliceSample(String project, String modelId, String evaluationId)
      throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      for (ModelEvaluationSlice modelEvaluationSlice :
          modelServiceClient.listModelEvaluationSlices(modelEvaluationName).iterateAll()) {
        System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
        System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
        System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
        System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

        Slice slice = modelEvaluationSlice.getSlice();
        System.out.format("Slice Dimensions: %s\n", slice.getDimension());
        System.out.format("Slice Value: %s\n\n", slice.getValue());
      }
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function listModelEvaluationSlices() {
  // Configure the parent resources
  const parent = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    parent,
  };

  // Get and print out a list of all the evaluation slices for this resource
  const [response] = await modelServiceClient.listModelEvaluationSlices(
    request
  );
  console.log('List model evaluation response', response);
  console.log(response);
}
listModelEvaluationSlices();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def list_model_evaluation_slices_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    parent = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.list_model_evaluation_slices(parent=parent)
    for model_evaluation_slice in response:
        print("model_evaluation_slice:", model_evaluation_slice)

对象检测

对于边界框指标，Vertex AI 会返回一组采用不同 IoU 阈值（介于 0 和 1 之间）和置信度阈值（介于 0 和 1 之间）的指标值。例如，您可以采用 IoU 阈值 0.85 和置信度阈值 0.8228 来缩小评估指标范围。通过查看这些不同的阈值，您可以看到它们如何影响其他指标，例如精确率和召回率。

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：模型所在的区域。例如 us-central1。
• PROJECT：您的项目 ID。
• MODEL_ID：您的模型的 ID。
• EVALUATION_ID：包含要列出评估切片的模型评估的 ID。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluationSlices": [
    {
      "name": "projects/693884908213/locations/us-central1/models/9199094820113481728/evaluations/3223746102406676480/slices/466270444805616887",
      "slice": {
        "dimension": "annotationSpec",
        "value": "Tomato"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/image_object_detection_metrics_1.0.0.yaml",
      "metrics": {
        "evaluatedBoundingBoxCount": 98,
        "boundingBoxMetrics": [
          {
            "iouThreshold": 0.75,
            "meanAveragePrecision": 0.22579378,
            "confidenceMetrics": [
              {
                "confidenceThreshold": 8.980461e-06,
                "recall": 0.3469388,
                "precision": 0.15525115,
                "f1Score": 0.21451104
              },
              {
                "confidenceThreshold": 0.10109353,
                "recall": 0.33673468,
                "precision": 0.35106382,
                "f1Score": 0.34374997
              },
              {
                "confidenceThreshold": 0.112416565,
                "recall": 0.3265306,
                "precision": 0.35555556,
                "f1Score": 0.34042552
              },
              ...
            ]
          },
          {
            "iouThreshold": 0.8,
            "meanAveragePrecision": 0.14461963,
            "confidenceMetrics": [
              {
                "confidenceThreshold": 8.980461e-06,
                "recall": 0.25510204,
                "precision": 0.11415525,
                "f1Score": 0.15772872
              },
              {
                "confidenceThreshold": 0.11577999,
                "recall": 0.24489796,
                "precision": 0.27272728,
                "f1Score": 0.25806454
              },
              {
                "confidenceThreshold": 0.1570652,
                "recall": 0.23469388,
                "precision": 0.28395063,
                "f1Score": 0.25698322
              },
              ...
            ]
          }
        ],
        "boundingBoxMeanAveragePrecision": 0.35238627
      },
      "createTime": "2020-10-09T23:35:43.654861Z"
    },
    {
      "name": "projects/693884908213/locations/us-central1/models/9199094820113481728/evaluations/3223746102406676480/slices/2285744162687356072",
      "slice": {
        "dimension": "annotationSpec",
        "value": "Cheese"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/image_object_detection_metrics_1.0.0.yaml",
      "metrics": {
        "evaluatedBoundingBoxCount": 46,
        "boundingBoxMetrics": [
          {
            "iouThreshold": 0.85,
            "meanAveragePrecision": 0.0110351965,
            "confidenceMetrics": [
              {
                "confidenceThreshold": 1.1757337e-05,
                "recall": 0.08695652,
                "precision": 0.019607844,
                "f1Score": 0.032
              },
              {
                "confidenceThreshold": 0.32720456,
                "recall": 0.06521739,
                "precision": 0.125,
                "f1Score": 0.08571429
              },
              {
                "confidenceThreshold": 0.4676356,
                "recall": 0.04347826,
                "precision": 0.1,
                "f1Score": 0.060606066
              },
              ...
            ]
          }
        ],
        "boundingBoxMeanAveragePrecision": 0.3591522
      },
      "createTime": "2020-10-09T23:35:43.661512Z"
    }
  ]
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class ListModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    listModelEvaluationSliceSample(project, modelId, evaluationId);
  }

  static void listModelEvaluationSliceSample(String project, String modelId, String evaluationId)
      throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      for (ModelEvaluationSlice modelEvaluationSlice :
          modelServiceClient.listModelEvaluationSlices(modelEvaluationName).iterateAll()) {
        System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
        System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
        System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
        System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

        Slice slice = modelEvaluationSlice.getSlice();
        System.out.format("Slice Dimensions: %s\n", slice.getDimension());
        System.out.format("Slice Value: %s\n\n", slice.getValue());
      }
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function listModelEvaluationSlices() {
  // Configure the parent resources
  const parent = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    parent,
  };

  // Get and print out a list of all the evaluation slices for this resource
  const [response] = await modelServiceClient.listModelEvaluationSlices(
    request
  );
  console.log('List model evaluation response', response);
  console.log(response);
}
listModelEvaluationSlices();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def list_model_evaluation_slices_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    parent = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.list_model_evaluation_slices(parent=parent)
    for model_evaluation_slice in response:
        print("model_evaluation_slice:", model_evaluation_slice)

表格

在下面选择您的目标对应的标签页：

分类

Vertex AI 会返回一组置信度指标。每个元素显示不同 confidenceThreshold 值的评估指标（从 0 开始，一直到 1）。通过查看不同的阈值，您可以看到阈值如何影响其他指标，例如精确率和召回率。

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：模型所在的区域。例如 us-central1。
• PROJECT：您的项目 ID。
• MODEL_ID：您的模型的 ID。
• EVALUATION_ID：包含要列出评估切片的模型评估的 ID。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluationSlices": [
    {
      "name": "projects/693884908213/locations/us-central1/models/705305922892726272/evaluations/4515484958386859492/slices/1785244630562158241",
      "slice": {
        "dimension": "annotationSpec",
        "value": "2"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 0.6108714,
        "auRoc": 0.9362428,
        "logLoss": 0.9680687,
        "confidenceMetrics": [
          {
            "recall": 1,
            "precision": 0.11417058,
            "falsePositiveRate": 1,
            "f1Score": 0.20494273,
            "recallAt1": 0.47843137,
            "precisionAt1": 0.6354167,
            "falsePositiveRateAt1": 0.035380337,
            "f1ScoreAt1": 0.5458613,
            "truePositiveCount": "510",
            "falsePositiveCount": "3957"
          },
          {
            "confidenceThreshold": 0.003269856,
            "recall": 0.9980392,
            "precision": 0.15108341,
            "falsePositiveRate": 0.7227698,
            "f1Score": 0.26243877,
            "recallAt1": 0.47843137,
            "precisionAt1": 0.6354167,
            "falsePositiveRateAt1": 0.035380337,
            "f1ScoreAt1": 0.5458613,
            "truePositiveCount": "509",
            "falsePositiveCount": "2860",
            "falseNegativeCount": "1",
            "trueNegativeCount": "1097"
          },
          {
            "confidenceThreshold": 0.016592776,
            "recall": 0.9882353,
            "precision": 0.23344141,
            "falsePositiveRate": 0.41824615,
            "f1Score": 0.37766954,
            "recallAt1": 0.47843137,
            "precisionAt1": 0.6354167,
            "falsePositiveRateAt1": 0.035380337,
            "f1ScoreAt1": 0.5458613,
            "truePositiveCount": "504",
            "falsePositiveCount": "1655",
            "falseNegativeCount": "6",
            "trueNegativeCount": "2302"
          },
          ...
        ]
      },
      "createTime": "2020-10-09T00:19:15.480435Z"
    },
    {
      "name": "projects/693884908213/locations/us-central1/models/705305922892726272/evaluations/4515484958386859492/slices/8107013027312442123",
      "slice": {
        "dimension": "annotationSpec",
        "value": "1"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 0.9916441,
        "auRoc": 0.93830043,
        "logLoss": 0.09145534,
        "confidenceMetrics": [
          {
            "recall": 1,
            "precision": 0.8858294,
            "falsePositiveRate": 1,
            "f1Score": 0.93945867,
            "recallAt1": 0.96461964,
            "precisionAt1": 0.9348518,
            "falsePositiveRateAt1": 0.52156866,
            "f1ScoreAt1": 0.94950247,
            "truePositiveCount": "3957",
            "falsePositiveCount": "510"
          },
          {
            "confidenceThreshold": 0.064618945,
            "recall": 0.9997473,
            "precision": 0.88639927,
            "falsePositiveRate": 0.9941176,
            "f1Score": 0.93966746,
            "recallAt1": 0.96461964,
            "precisionAt1": 0.9348518,
            "falsePositiveRateAt1": 0.52156866,
            "f1ScoreAt1": 0.94950247,
            "truePositiveCount": "3956",
            "falsePositiveCount": "507",
            "falseNegativeCount": "1",
            "trueNegativeCount": "3"
          },
          ...

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class ListModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    listModelEvaluationSliceSample(project, modelId, evaluationId);
  }

  static void listModelEvaluationSliceSample(String project, String modelId, String evaluationId)
      throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      for (ModelEvaluationSlice modelEvaluationSlice :
          modelServiceClient.listModelEvaluationSlices(modelEvaluationName).iterateAll()) {
        System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
        System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
        System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
        System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

        Slice slice = modelEvaluationSlice.getSlice();
        System.out.format("Slice Dimensions: %s\n", slice.getDimension());
        System.out.format("Slice Value: %s\n\n", slice.getValue());
      }
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function listModelEvaluationSlices() {
  // Configure the parent resources
  const parent = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    parent,
  };

  // Get and print out a list of all the evaluation slices for this resource
  const [response] = await modelServiceClient.listModelEvaluationSlices(
    request
  );
  console.log('List model evaluation response', response);
  console.log(response);
}
listModelEvaluationSlices();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def list_model_evaluation_slices_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    parent = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.list_model_evaluation_slices(parent=parent)
    for model_evaluation_slice in response:
        print("model_evaluation_slice:", model_evaluation_slice)

预测

表格预测模型没有评估指标切片。

回归

表格回归模型没有评估指标切片。

文本

在下面选择您的目标对应的标签页：

分类

Vertex AI 会返回一组置信度指标。每个元素显示不同 confidenceThreshold 值的评估指标（从 0 开始，一直到 1）。通过查看不同的阈值，您可以看到阈值如何影响其他指标，例如精确率和召回率。

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：模型所在的区域。例如 us-central1。
• PROJECT：您的项目 ID。
• MODEL_ID：您的模型的 ID。
• EVALUATION_ID：包含要列出评估切片的模型评估的 ID。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluationSlices": [
    {
      "name": "projects/693884908213/locations/us-central1/models/4597084504010522624/evaluations/5327525262714208256/slices/28418181714804292",
      "slice": {
        "dimension": "annotationSpec",
        "value": "leisure"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 0.8596267,
        "logLoss": 0.07142017,
        "confidenceMetrics": [
          {
            "recall": 1,
            "precision": 0.05822785,
            "f1Score": 0.11004785
          },
          {
            "confidenceThreshold": 0.05,
            "recall": 0.95652175,
            "precision": 0.3882353,
            "f1Score": 0.5523013
          },
          {
            "confidenceThreshold": 0.1,
            "recall": 0.89855075,
            "precision": 0.53913045,
            "f1Score": 0.67391306
          },
          ...
        ]
      },
      "createTime": "2020-10-29T01:17:39.368890Z"
    },
    {
      "name": "projects/693884908213/locations/us-central1/models/4597084504010522624/evaluations/5327525262714208256/slices/177673261689801402",
      "slice": {
        "dimension": "annotationSpec",
        "value": "achievement"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 0.95337653,
        "logLoss": 0.17853451,
        "confidenceMetrics": [
          {
            "recall": 1,
            "precision": 0.31814346,
            "f1Score": 0.48271447
          },
          {
            "confidenceThreshold": 0.05,
            "recall": 0.9920424,
            "precision": 0.70433146,
            "f1Score": 0.8237885
          },
          ...

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class ListModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    listModelEvaluationSliceSample(project, modelId, evaluationId);
  }

  static void listModelEvaluationSliceSample(String project, String modelId, String evaluationId)
      throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      for (ModelEvaluationSlice modelEvaluationSlice :
          modelServiceClient.listModelEvaluationSlices(modelEvaluationName).iterateAll()) {
        System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
        System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
        System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
        System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

        Slice slice = modelEvaluationSlice.getSlice();
        System.out.format("Slice Dimensions: %s\n", slice.getDimension());
        System.out.format("Slice Value: %s\n\n", slice.getValue());
      }
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function listModelEvaluationSlices() {
  // Configure the parent resources
  const parent = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    parent,
  };

  // Get and print out a list of all the evaluation slices for this resource
  const [response] = await modelServiceClient.listModelEvaluationSlices(
    request
  );
  console.log('List model evaluation response', response);
  console.log(response);
}
listModelEvaluationSlices();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def list_model_evaluation_slices_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    parent = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.list_model_evaluation_slices(parent=parent)
    for model_evaluation_slice in response:
        print("model_evaluation_slice:", model_evaluation_slice)

实体提取

Vertex AI 会返回一组置信度指标。每个元素显示不同 confidenceThreshold 值的评估指标（从 0 开始，一直到 1）。通过查看不同的阈值，您可以看到阈值如何影响其他指标，例如精确率和召回率。

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：模型所在的区域。例如 us-central1。
• PROJECT：您的项目 ID。
• MODEL_ID：您的模型的 ID。
• EVALUATION_ID：包含要列出评估切片的模型评估的 ID。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluationSlices": [
    {
      "name": "projects/693884908213/locations/us-central1/models/6188544022332571648/evaluations/6791195141609619456/slices/381924577273410860",
      "slice": {
        "dimension": "annotationSpec",
        "value": "CompositeMention"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/text_extraction_metrics_1.0.0.yaml",
      "metrics": {
        "confidenceMetrics": [
          {
            "confidenceThreshold": 0.04,
            "recall": 0.6666667,
            "precision": 0.6666667,
            "f1Score": 0.6666667
          },
          {
            "confidenceThreshold": 0.96,
            "recall": 0.6666667,
            "precision": 0.85714287,
            "f1Score": 0.75
          },
          {
            "confidenceThreshold": 0.15,
            "recall": 0.6666667,
            "precision": 0.6666667,
            "f1Score": 0.6666667
          },
          ...
        ]
      },
      "createTime": "2020-10-28T23:43:59.713493Z"
    },
    {
      "name": "projects/693884908213/locations/us-central1/models/6188544022332571648/evaluations/6791195141609619456/slices/5211460974948430734",
      "slice": {
        "dimension": "annotationSpec",
        "value": "DiseaseClass"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/text_extraction_metrics_1.0.0.yaml",
      "metrics": {
        "confidenceMetrics": [
          {
            "confidenceThreshold": 0.04,
            "recall": 0.6545454,
            "precision": 0.5625,
            "f1Score": 0.605042
          },
          {
            "confidenceThreshold": 0.96,
            "recall": 0.56363636,
            "precision": 0.75609756,
            "f1Score": 0.6458333
          },
          ...

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class ListModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    listModelEvaluationSliceSample(project, modelId, evaluationId);
  }

  static void listModelEvaluationSliceSample(String project, String modelId, String evaluationId)
      throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      for (ModelEvaluationSlice modelEvaluationSlice :
          modelServiceClient.listModelEvaluationSlices(modelEvaluationName).iterateAll()) {
        System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
        System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
        System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
        System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

        Slice slice = modelEvaluationSlice.getSlice();
        System.out.format("Slice Dimensions: %s\n", slice.getDimension());
        System.out.format("Slice Value: %s\n\n", slice.getValue());
      }
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function listModelEvaluationSlices() {
  // Configure the parent resources
  const parent = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    parent,
  };

  // Get and print out a list of all the evaluation slices for this resource
  const [response] = await modelServiceClient.listModelEvaluationSlices(
    request
  );
  console.log('List model evaluation response', response);
  console.log(response);
}
listModelEvaluationSlices();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def list_model_evaluation_slices_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    parent = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.list_model_evaluation_slices(parent=parent)
    for model_evaluation_slice in response:
        print("model_evaluation_slice:", model_evaluation_slice)

情感分析

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：模型所在的区域。例如 us-central1。
• PROJECT：您的项目 ID。
• MODEL_ID：您的模型的 ID。
• EVALUATION_ID：包含要列出评估切片的模型评估的 ID。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluationSlices": [
    {
      "name": "projects/693884908213/locations/us-central1/models/5625594068911259648/evaluations/6055278814999609344/slices/224427761824380763",
      "slice": {
        "dimension": "annotationSpec",
        "value": "0"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/text_sentiment_metrics_1.0.0.yaml",
      "metrics": {
        "precision": 1
      },
      "createTime": "2020-10-29T00:43:30.509069Z"
    },
    {
      "name": "projects/693884908213/locations/us-central1/models/5625594068911259648/evaluations/6055278814999609344/slices/3809169538370653981",
      "slice": {
        "dimension": "annotationSpec",
        "value": "6"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/text_sentiment_metrics_1.0.0.yaml",
      "metrics": {
        "precision": 1
      },
      "createTime": "2020-10-29T00:43:30.529540Z"
    },
    {
      "name": "projects/693884908213/locations/us-central1/models/5625594068911259648/evaluations/6055278814999609344/slices/4398720854770784612",
      "slice": {
        "dimension": "annotationSpec",
        "value": "2"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/text_sentiment_metrics_1.0.0.yaml",
      "metrics": {
        "precision": 0.3548387,
        "recall": 0.47826087,
        "f1Score": 0.4074074
      },
      "createTime": "2020-10-29T00:43:30.516171Z"
    },
    ...

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class ListModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    listModelEvaluationSliceSample(project, modelId, evaluationId);
  }

  static void listModelEvaluationSliceSample(String project, String modelId, String evaluationId)
      throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      for (ModelEvaluationSlice modelEvaluationSlice :
          modelServiceClient.listModelEvaluationSlices(modelEvaluationName).iterateAll()) {
        System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
        System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
        System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
        System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

        Slice slice = modelEvaluationSlice.getSlice();
        System.out.format("Slice Dimensions: %s\n", slice.getDimension());
        System.out.format("Slice Value: %s\n\n", slice.getValue());
      }
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function listModelEvaluationSlices() {
  // Configure the parent resources
  const parent = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    parent,
  };

  // Get and print out a list of all the evaluation slices for this resource
  const [response] = await modelServiceClient.listModelEvaluationSlices(
    request
  );
  console.log('List model evaluation response', response);
  console.log(response);
}
listModelEvaluationSlices();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def list_model_evaluation_slices_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    parent = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.list_model_evaluation_slices(parent=parent)
    for model_evaluation_slice in response:
        print("model_evaluation_slice:", model_evaluation_slice)

视频

在下面选择您的目标对应的标签页：

动作识别

Vertex AI 会返回一系列视频操作识别指标。每个元素显示不同 precisionWindowLength 和 confidenceThreshold 值的评估指标。通过查看不同窗口长度和置信度阈值的评估指标，您可以看到它们如何影响其他指标，例如精确率和召回率。

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：模型所在的区域。例如 us-central1。
• PROJECT：您的项目 ID。
• MODEL_ID：您的模型的 ID。
• EVALUATION_ID：包含要列出评估切片的模型评估的 ID。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluationSlices": [
    {
      "name": "projects/693884908213/locations/us-central1/models/2108872098167390208/evaluations/2501984888492654592/slices/600455355983391492",
      "slice": {
        "dimension": "annotationSpec",
        "value": "run"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/video_action_recognition_metrics_1.0.0.yaml",
      "metrics": {
        "evaluatedActionCount": 4,
        "videoActionMetrics": [
          {
            "precisionWindowLength": "1s",
            "meanAveragePrecision": 1,
            "confidenceMetrics": [
              {
                "confidenceThreshold": 0.8158053,
                "recall": 0.5,
                "precision": 1,
                "f1Score": 0.6666667
              },
              {
                "confidenceThreshold": 0.793864,
                "recall": 1,
                "precision": 1,
                "f1Score": 1
              },
              {
                "confidenceThreshold": 0.13212349,
                "recall": 1,
                "precision": 0.6666667,
                "f1Score": 0.8
              },
              {
                "confidenceThreshold": 0.07055229,
                "recall": 1,
                "precision": 0.5,
                "f1Score": 0.6666667
              }
            ]
          },
          {
            "precisionWindowLength": "2s",
            "meanAveragePrecision": 1,
            "confidenceMetrics": [
              {
                "confidenceThreshold": 0.8158053,
                "recall": 0.5,
                "precision": 1,
                "f1Score": 0.6666667
              },
              {
                "confidenceThreshold": 0.793864,
                "recall": 1,
                "precision": 1,
                "f1Score": 1
              },
              ...
            ]
          }
        ]
      },
      "createTime": "2020-11-09T20:02:06.010615Z"
    },
    {
      "name": "projects/693884908213/locations/us-central1/models/2108872098167390208/evaluations/2501984888492654592/slices/7967805175608198137",
      "slice": {
        "dimension": "annotationSpec",
        "value": "swim"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/video_action_recognition_metrics_1.0.0.yaml",
      "metrics": {
        "evaluatedActionCount": 4,
        "videoActionMetrics": [
          {
            "precisionWindowLength": "1s",
            "meanAveragePrecision": 1,
            "confidenceMetrics": [
              {
                "confidenceThreshold": 0.72200835,
                "recall": 0.5,
                "precision": 1,
                "f1Score": 0.6666667
              },
              {
                "confidenceThreshold": 0.5425852,
                "recall": 1,
                "precision": 1,
                "f1Score": 1
              },
              {
                "confidenceThreshold": 0.06143289,
                "recall": 1,
                "precision": 0.6666667,
                "f1Score": 0.8
              },
              ...
            ]
          }
        ]
      },
      "createTime": "2020-11-09T20:02:06.007733Z"
    }
  ]
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class ListModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    listModelEvaluationSliceSample(project, modelId, evaluationId);
  }

  static void listModelEvaluationSliceSample(String project, String modelId, String evaluationId)
      throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      for (ModelEvaluationSlice modelEvaluationSlice :
          modelServiceClient.listModelEvaluationSlices(modelEvaluationName).iterateAll()) {
        System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
        System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
        System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
        System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

        Slice slice = modelEvaluationSlice.getSlice();
        System.out.format("Slice Dimensions: %s\n", slice.getDimension());
        System.out.format("Slice Value: %s\n\n", slice.getValue());
      }
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function listModelEvaluationSlices() {
  // Configure the parent resources
  const parent = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    parent,
  };

  // Get and print out a list of all the evaluation slices for this resource
  const [response] = await modelServiceClient.listModelEvaluationSlices(
    request
  );
  console.log('List model evaluation response', response);
  console.log(response);
}
listModelEvaluationSlices();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def list_model_evaluation_slices_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    parent = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.list_model_evaluation_slices(parent=parent)
    for model_evaluation_slice in response:
        print("model_evaluation_slice:", model_evaluation_slice)

分类

Vertex AI 会返回一组置信度指标。每个元素显示不同 confidenceThreshold 值的评估指标（从 0 开始，一直到 1）。通过查看不同的阈值，您可以看到阈值如何影响其他指标，例如精确率和召回率。

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：模型所在的区域。例如 us-central1。
• PROJECT：您的项目 ID。
• MODEL_ID：您的模型的 ID。
• EVALUATION_ID：包含要列出评估切片的模型评估的 ID。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluationSlices": [
    {
      "name": "projects/693884908213/locations/us-central1/models/1335809870724595712/evaluations/4193145922198175744/slices/2184781668104570610",
      "slice": {
        "dimension": "annotationSpec",
        "value": "pullup"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 1,
        "confidenceMetrics": [
          {
            "confidenceThreshold": 0.103617236,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          {
            "confidenceThreshold": 0.104728825,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          {
            "confidenceThreshold": 0.10573554,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          ...
        ]
      },
      "createTime": "2020-10-09T01:43:47.188017Z"
    },
    {
      "name": "projects/693884908213/locations/us-central1/models/1335809870724595712/evaluations/4193145922198175744/slices/3547424436739863990",
      "slice": {
        "dimension": "annotationSpec",
        "value": "golf"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 1,
        "confidenceMetrics": [
          {
            "confidenceThreshold": 0.103617236,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          {
            "confidenceThreshold": 0.104728825,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          {
            "confidenceThreshold": 0.10573554,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          ...

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class ListModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    listModelEvaluationSliceSample(project, modelId, evaluationId);
  }

  static void listModelEvaluationSliceSample(String project, String modelId, String evaluationId)
      throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      for (ModelEvaluationSlice modelEvaluationSlice :
          modelServiceClient.listModelEvaluationSlices(modelEvaluationName).iterateAll()) {
        System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
        System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
        System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
        System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

        Slice slice = modelEvaluationSlice.getSlice();
        System.out.format("Slice Dimensions: %s\n", slice.getDimension());
        System.out.format("Slice Value: %s\n\n", slice.getValue());
      }
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function listModelEvaluationSlices() {
  // Configure the parent resources
  const parent = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    parent,
  };

  // Get and print out a list of all the evaluation slices for this resource
  const [response] = await modelServiceClient.listModelEvaluationSlices(
    request
  );
  console.log('List model evaluation response', response);
  console.log(response);
}
listModelEvaluationSlices();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def list_model_evaluation_slices_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    parent = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.list_model_evaluation_slices(parent=parent)
    for model_evaluation_slice in response:
        print("model_evaluation_slice:", model_evaluation_slice)

对象跟踪

对于边界框指标，Vertex AI 会返回一组采用不同 IoU 阈值（介于 0 和 1 之间）和置信度阈值（介于 0 和 1 之间）的指标值。例如，您可以采用 IoU 阈值 0.85 和置信度阈值 0.8228 来缩小评估指标范围。通过查看这些不同的阈值，您可以看到它们如何影响其他指标，例如精确率和召回率。

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：模型所在的区域。例如 us-central1。
• PROJECT：您的项目 ID。
• MODEL_ID：您的模型的 ID。
• EVALUATION_ID：包含要列出评估切片的模型评估的 ID。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluationSlices": [
    {
      "name": "projects/693884908213/locations/us-central1/models/5582326087335018496/evaluations/1322661910679650304/slices/4253049326429486016",
      "slice": {
        "dimension": "annotationSpec",
        "value": "large_veh_bus"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/video_object_tracking_metrics_1.0.0.yaml",
      "metrics": {
        "boundingBoxMetrics": [
          {
            "iouThreshold": 0.5,
            "meanAveragePrecision": 0.6668269,
            "confidenceMetrics": [
              {
                "recall": 1,
                "precision": 0.8125,
                "f1Score": 0.8965517
              },
              {
                "confidenceThreshold": 0.29,
                "recall": 0.15384616,
                "precision": 0.4,
                "f1Score": 0.22222222
              },
              {
                "confidenceThreshold": 0.556075,
                "recall": 0.15384616,
                "precision": 1,
                "f1Score": 0.26666668
              },
              ...
            ]
          }
        ],
        "boundingBoxMeanAveragePrecision": 0.6668269
      },
      "createTime": "2020-10-12T22:42:59.866314Z"
    },
    {
      "name": "projects/693884908213/locations/us-central1/models/5582326087335018496/evaluations/1322661910679650304/slices/6351049261036277514",
      "slice": {
        "dimension": "annotationSpec",
        "value": "pickup_suv_van"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/video_object_tracking_metrics_1.0.0.yaml",
      "metrics": {
        "boundingBoxMetrics": [
          {
            "iouThreshold": 0.5,
            "meanAveragePrecision": 0.5899872,
            "confidenceMetrics": [
              {
                "recall": 1,
                "precision": 0.71428573,
                "f1Score": 0.8333333
              },
              {
                "confidenceThreshold": 0.29,
                "recall": 0.2,
                "precision": 0.33333334,
                "f1Score": 0.25
              },
              {
                "confidenceThreshold": 0.3687303,
                "recall": 0.16363636,
                "precision": 0.75,
                "f1Score": 0.26865673
              },
              {
                "confidenceThreshold": 0.52331626,
                "recall": 0.14545454,
                "precision": 0.72727275,
                "f1Score": 0.24242425
              },
              ...

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class ListModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    listModelEvaluationSliceSample(project, modelId, evaluationId);
  }

  static void listModelEvaluationSliceSample(String project, String modelId, String evaluationId)
      throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      for (ModelEvaluationSlice modelEvaluationSlice :
          modelServiceClient.listModelEvaluationSlices(modelEvaluationName).iterateAll()) {
        System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
        System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
        System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
        System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

        Slice slice = modelEvaluationSlice.getSlice();
        System.out.format("Slice Dimensions: %s\n", slice.getDimension());
        System.out.format("Slice Value: %s\n\n", slice.getValue());
      }
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function listModelEvaluationSlices() {
  // Configure the parent resources
  const parent = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    parent,
  };

  // Get and print out a list of all the evaluation slices for this resource
  const [response] = await modelServiceClient.listModelEvaluationSlices(
    request
  );
  console.log('List model evaluation response', response);
  console.log(response);
}
listModelEvaluationSlices();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def list_model_evaluation_slices_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    parent = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.list_model_evaluation_slices(parent=parent)
    for model_evaluation_slice in response:
        print("model_evaluation_slice:", model_evaluation_slice)

获取单个切片的指标

如需查看单个切片的评估指标，请使用 projects.locations.models.evaluations.slices.get 方法。您必须拥有切片 ID（该 ID 会在您列出所有切片时提供）。以下示例适用于所有数据类型和目标。

REST 和命令行

在使用任何请求数据之前，请先进行以下替换：

• LOCATION：模型所在的区域。例如 us-central1。
• PROJECT：您的项目 ID。
• MODEL_ID：您的模型的 ID。
• EVALUATION_ID：包含要检索的评估切片的模型评估 ID。
• SLICE_ID：要获取的评估切片的 ID。
• PROJECT_NUMBER：您的项目编号。
• EVALUATION_METRIC_SCHEMA_FILE_NAME：定义要返回的评估指标的架构文件名称，例如 classification_metrics_1.0.0。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID

如需发送请求，请选择以下方式之一：

curl

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

curl -X GET \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID"

PowerShell

注意：请确保已将 GOOGLE_APPLICATION_CREDENTIALS 环境变量设置为服务帐号私钥文件的路径。

执行以下命令：

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID",
  "slice": {
    "dimension": "annotationSpec",
    "value": "a particular class or label"
  },
  "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/EVALUATION_METRIC_SCHEMA_FILE_NAME.yaml",
  "metrics": {
    evaluation metrics for the slice
  },
  "createTime": "2020-10-08T23:35:54.770876Z"
}

Java

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Java API 参考文档。

在 GitHub 上查看 反馈

import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSliceName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    String sliceId = "YOUR_SLICE_ID";
    getModelEvaluationSliceSample(project, modelId, evaluationId, sliceId);
  }

  static void getModelEvaluationSliceSample(
      String project, String modelId, String evaluationId, String sliceId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationSliceName modelEvaluationSliceName =
          ModelEvaluationSliceName.of(project, location, modelId, evaluationId, sliceId);

      ModelEvaluationSlice modelEvaluationSlice =
          modelServiceClient.getModelEvaluationSlice(modelEvaluationSliceName);

      System.out.println("Get Model Evaluation Slice Response");
      System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
      System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
      System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
      System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

      Slice slice = modelEvaluationSlice.getSlice();
      System.out.format("Slice Dimensions: %s\n", slice.getDimension());
      System.out.format("Slice Value: %s\n", slice.getValue());
    }
  }
}

Node.js

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

在 GitHub 上查看 反馈

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const sliceId = 'YOUR_SLICE_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');
// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};
// Specifies the location of the api endpoint
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationSlice() {
  // Configure the parent resource
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}/slices/${sliceId}`;
  const request = {
    name,
  };

  // Get and print out a list of all the endpoints for this resource
  const [response] = await modelServiceClient.getModelEvaluationSlice(
    request
  );

  console.log('Get model evaluation slice');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics_Schema_Uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);

  console.log('Slice');
  const slice = response.slice;
  console.log(`\tDimension :${slice.dimension}`);
  console.log(`\tValue :${slice.value}`);
}
getModelEvaluationSlice();

Python

如需了解如何安装和使用 Vertex AI 客户端库，请参阅 Vertex AI 客户端库。如需了解详情，请参阅 Vertex AI Python API 参考文档。

在 GitHub 上查看 反馈

from google.cloud import aiplatform

def get_model_evaluation_slice_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    slice_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_slice_path(
        project=project,
        location=location,
        model=model_id,
        evaluation=evaluation_id,
        slice=slice_id,
    )
    response = client.get_model_evaluation_slice(name=name)
    print("response:", response)

迭代模型

当模型未达到您的预期时，模型评估指标可为您提供调试模型的起点。例如，低精确率和召回率可能表明您的模型需要额外的训练数据或者存在不一致的标签。完美的精确率和召回率可能表明测试数据难以预测，并且可能无法有效泛化。

您可以迭代训练数据并创建新模型。创建新模型后，您可以比较现有模型和新模型的评估指标。

以下建议可帮助您改进为各项添加标签的模型（例如分类或检测模型）：

• 建议在您的训练数据中添加更多样本或范围更广的示例。例如，对于图片分类模型，您可以添加角度更宽的图片、分辨率更高或更低的图片或不同的视角。如需更多指导，请参阅针对特定数据类型和目标准备数据部分。
• 考虑移除没有大量样本的类或标签。样本不足会导致模型无法一致、自信地对这些类或标签进行预测。
• 机器无法解读类或标签的名称，也无法理解它们之间的细微差别，例如“door”和“door_with_knob”。您必须提供数据来帮助机器识别这些细微差别。
• 使用更多的真正例和真负例样本来扩充您的数据，尤其是靠近决策边界的样本，以降低模型混淆。
• 指定您自己的数据拆分（训练、验证和测试）。Vertex AI 会为每个集合随机分配项。因此，相似的重复项可能会分配在训练集和验证集中，这可能导致过拟合，进而造成测试集性能不佳。如需详细了解如何设置您自己的数据拆分，请参阅 AutoML 模型的数据拆分简介。
• 如果模型的评估指标包含混淆矩阵，您可以查看模型是否会混淆两个标签，其中模型预测特定标签明显比真实标签多。查看您的数据，并确保正确标记样本。
• 如果您的训练时间很短（较低的节点最大小时数），您可以延长训练时间来获得质量更高的模型（较高的节点数上限）。

表式数据

除了前面的建议外，您还可以尝试执行以下特定于标签页的建议，以在您的模型上迭代。对于性能较差的模型，请尝试以下建议：

• 从训练中排除任何不属于预测的列，例如 ID 列。
• 检查转换，确保所有列都设置为正确的转换。了解详情。
• 检查您的数据，并确保错误数不多。例如，不接受 null 值的列中缺失值会导致该行被忽略或估算。了解详情。
• 导出并检查测试数据集。 分析模型何时进行不正确的预测，以确定是否需要更多训练数据来获得特定结果，或者训练数据是否引入了泄露。
• 对于预测模型，请考虑执行以下操作：
  • 增加上下文窗口，从而增加模型的预测模式的回溯时间。如需了解详情，请参阅设置上下文窗口和预测范围的注意事项。
  • 在训练选项中，检查在预测时值不可用或可用的时间相关列是否已正确标记。
  • 在训练数据中，添加更多数据，这些数据包含要建模的行为的几个示例。例如，您可以训练一个与不同时间序列相关的模型，例如侵蚀效应（新产品降低了对现有产品的需求）或光环效应（其中新产品增加了相关产品的需求）。

对于具有完美性能的模型，请尝试以下建议：

• 检查是否存在目标泄露。目标泄露是指训练数据中包含有在训练时本应无法知晓的基于结果的特征。例如，如果您训练一个用于确定新用户是否会进行购买的模型，并在其中包括了“频繁买家”编号，则该模型的评估指标将极其优异，但在真实数据上的性能会很差，因为真实数据中不可能包括频繁买家编号。

  若要检查是否存在目标泄露，请查看模型的评估标签页上的特征重要性图。确保具有高重要性的列确实具有预测功能，并且不会泄露关于目标的信息。

• 对于分类和回归模型，如果数据的时间很重要，请确保使用时间列或基于时间的手动拆分。不这样做可能会导致您的评估指标出现偏差。如需了解详情，请参阅准备表训练数据页面上的时间列。

后续步骤

• 使用 Google Cloud 控制台部署模型。
• 使用 API 部署模型。
• 获取批量预测结果。