评估和迭代 AutoML 视频分类模型

本页面介绍了如何评估 AutoML 视频分类模型，以便您可以迭代模型。

Vertex AI 提供了模型评估指标，可帮助您确定模型的性能，例如精确率和召回率指标。Vertex AI 使用测试集来计算评估指标。

如何使用模型评估指标

模型评估指标提供模型在测试集上的表现的量化测量结果。如何解读和使用这些指标取决于您的业务需求和模型训练所要解决的问题。例如，假正例的容忍度可能低于假负例，反之亦然。这些问题会影响您重点关注的指标。

如需详细了解如何迭代模型以改善其性能，请参阅迭代模型。

Vertex AI 返回的评估指标

Vertex AI 会返回几个不同的评估指标，例如精确率、召回率和置信度阈值。Vertex AI 返回的指标取决于您的模型目标。例如，与图像对象检测模型相比，Vertex AI 为图像分类模型提供了不同的评估指标。

架构文件可从 Cloud Storage 位置下载，它决定了 Vertex AI 为每个目标提供的评估指标。以下架构文件描述了用于视频分类的评估指标。

您可以从以下 Cloud Storage 位置查看和下载架构文件：
gs://google-cloud-aiplatform/schema/modelevaluation/

AuPRC：精确率/召回率 (PR) 曲线下的面积，亦称为平均精度。此值的范围在 0 到 1 之间，值越大表示模型质量越高。
置信度阈值：用于确定要返回的预测的置信度分数。模型会返回不低于此值的预测。如果提高置信度阈值，精确率会相应地提高，但召回率会降低。Vertex AI 以不同的阈值返回置信度指标，以显示阈值如何影响精确率和召回率。
召回率：模型正确预测此类别的预测所占的比例。也称为真正例率。
精确率：模型生成的正确分类预测所占的比例。
F1 得分：精确率和召回率的调和平均数。F1 是一个很实用的指标，当您希望在精确率和召回率之间取得平衡，而类别分布又不均匀时，该指标会非常有帮助。
混淆矩阵：混淆矩阵显示模型正确预测结果的频率。如果预测结果不正确，矩阵会改为显示模型预测的内容。混淆矩阵可帮助您了解模型“混淆”两个结果的出现位置。

获取评估指标

您可以获取模型的一组总体评估指标，以及针对特定类或标签的评估指标。特定类或标签的评估指标也称为评估切片。以下内容介绍如何使用 Google Cloud 控制台或 API 获取总体评估指标和评估切片。

Google Cloud 控制台

在 Google Cloud 控制台的 Vertex AI 部分中，转到模型页面。

打开“模型”页面
在区域下拉列表中，选择您的模型所在的区域。
在模型列表中，点击您的模型以打开模型的评估标签页。

在评估标签页中，您可以查看模型的总体评估指标，例如平均精确率和召回率。

如果模型目标具有评估切片，则控制台会显示标签列表。您可以点击标签来查看该标签的评估指标，如以下示例所示：

API

对于每种数据类型和目标，获取评估指标的 API 请求是相同的，但输出有所不同。以下示例显示了相同的请求和不同的响应。

获取聚合模型评估指标

总体模型评估指标提供有关整个模型的信息。如需查看特定切片的相关信息，请列出模型评估切片。

如需查看总体模型评估指标，请使用 projects.locations.models.evaluations.get 方法。

Vertex AI 会返回一组置信度指标。每个元素显示不同 confidenceThreshold 值的评估指标（从 0 开始，一直到 1）。通过查看不同的阈值，您可以看到阈值如何影响其他指标，例如精确率和召回率。

选择与您的语言或环境对应的标签页：

REST

在使用任何请求数据之前，请先进行以下替换：

LOCATION：存储模型的区域。
PROJECT：您的项目 ID。
MODEL_ID：模型资源的 ID。
PROJECT_NUMBER：自动生成的项目编号。
EVALUATION_ID：模型评估的 ID（显示在响应中）。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI，或者使用了 Cloud Shell，这会使您自动登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

执行以下命令：

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

执行以下命令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 1,
        "confidenceMetrics": [
          {
            "confidenceThreshold": 0.103617236,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          {
            "confidenceThreshold": 0.104728825,
            "recall": 1,
            "precision": 0.2016129,
            "f1Score": 0.33557048
          },
          {
            "confidenceThreshold": 0.10573554,
            "recall": 1,
            "precision": 0.20283976,
            "f1Score": 0.3372681
          },
          ...
        ],
        "confusionMatrix": {
          "annotationSpecs": [
            {
              "id": "1004478638764064768",
              "displayName": "pullup"
            },
            {
              "id": "3310321647977758720",
              "displayName": "kick_ball"
            },
            {
              "id": "5616164657191452672",
              "displayName": "cartwheel"
            },
            {
              "id": "7922007666405146624",
              "displayName": "ride_horse"
            },
            {
              "id": "8498468418708570112",
              "displayName": "golf"
            }
          ],
          "rows": [
            [
              20,
              0,
              0,
              0,
              0
            ],
            [
              0,
              20,
              0,
              0,
              0
            ],
            [
              0,
              0,
              20,
              0,
              0
            ],
            [
              0,
              0,
              0,
              20,
              0
            ],
            [
              0,
              0,
              0,
              0,
              20
            ]
          ]
        }
      },
      "createTime": "2020-10-09T01:43:47.000284Z",
      "sliceDimensions": [
        "annotationSpec"
      ]
    }
  ]
}

Java

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Java 设置说明执行操作。如需了解详情，请参阅 Vertex AI Java API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。


import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationVideoClassificationSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    getModelEvaluationVideoClassification(project, modelId, evaluationId);
  }

  static void getModelEvaluationVideoClassification(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Video Classification Response");
      System.out.format("Name: %s\n", modelEvaluation.getName());
      System.out.format("Metrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("Metrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("Create Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("Slice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Node.js 设置说明执行操作。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationVideoClassification() {
  // Configure the parent resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Create get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation video classification response');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);
  console.log(`\tSlice dimensions : ${response.sliceDimensions}`);
}
getModelEvaluationVideoClassification();

Python

如需了解如何安装或更新 Python，请参阅安装 Python 版 Vertex AI SDK。如需了解详情，请参阅 Python API 参考文档。

from google.cloud import aiplatform

def get_model_evaluation_video_classification_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

列出所有评估切片

projects.locations.models.evaluations.slices.list 方法可列出您的模型的所有评估切片。您必须拥有模型的评估 ID，才能在查看总体评估指标时获得此 ID。

您可以使用模型评估切片来确定模型在特定标签上的表现。value 字段指示与指标对应的标签。

REST

在使用任何请求数据之前，请先进行以下替换：

LOCATION：模型所在的区域。例如 us-central1。
PROJECT：您的项目 ID。
MODEL_ID：您的模型的 ID。
EVALUATION_ID：包含要列出评估切片的模型评估的 ID。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices

如需发送请求，请选择以下方式之一：

curl

执行以下命令：

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices"

PowerShell

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

执行以下命令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluationSlices": [
    {
      "name": "projects/693884908213/locations/us-central1/models/1335809870724595712/evaluations/4193145922198175744/slices/2184781668104570610",
      "slice": {
        "dimension": "annotationSpec",
        "value": "pullup"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 1,
        "confidenceMetrics": [
          {
            "confidenceThreshold": 0.103617236,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          {
            "confidenceThreshold": 0.104728825,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          {
            "confidenceThreshold": 0.10573554,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          ...
        ]
      },
      "createTime": "2020-10-09T01:43:47.188017Z"
    },
    {
      "name": "projects/693884908213/locations/us-central1/models/1335809870724595712/evaluations/4193145922198175744/slices/3547424436739863990",
      "slice": {
        "dimension": "annotationSpec",
        "value": "golf"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 1,
        "confidenceMetrics": [
          {
            "confidenceThreshold": 0.103617236,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          {
            "confidenceThreshold": 0.104728825,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          {
            "confidenceThreshold": 0.10573554,
            "recall": 1,
            "precision": 0.2,
            "f1Score": 0.33333334
          },
          ...

Java

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Java 设置说明执行操作。如需了解详情，请参阅 Vertex AI Java API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。


import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class ListModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    listModelEvaluationSliceSample(project, modelId, evaluationId);
  }

  static void listModelEvaluationSliceSample(String project, String modelId, String evaluationId)
      throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      for (ModelEvaluationSlice modelEvaluationSlice :
          modelServiceClient.listModelEvaluationSlices(modelEvaluationName).iterateAll()) {
        System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
        System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
        System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
        System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

        Slice slice = modelEvaluationSlice.getSlice();
        System.out.format("Slice Dimensions: %s\n", slice.getDimension());
        System.out.format("Slice Value: %s\n\n", slice.getValue());
      }
    }
  }
}

Node.js

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Node.js 设置说明执行操作。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function listModelEvaluationSlices() {
  // Configure the parent resources
  const parent = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    parent,
  };

  // Get and print out a list of all the evaluation slices for this resource
  const [response] =
    await modelServiceClient.listModelEvaluationSlices(request);
  console.log('List model evaluation response', response);
  console.log(response);
}
listModelEvaluationSlices();

Python

如需了解如何安装或更新 Python，请参阅安装 Python 版 Vertex AI SDK。如需了解详情，请参阅 Python API 参考文档。

from google.cloud import aiplatform

def list_model_evaluation_slices_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    parent = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.list_model_evaluation_slices(parent=parent)
    for model_evaluation_slice in response:
        print("model_evaluation_slice:", model_evaluation_slice)

获取单个切片的指标

如需查看单个切片的评估指标，请使用 projects.locations.models.evaluations.slices.get 方法。您必须拥有切片 ID（该 ID 会在您列出所有切片时提供）。以下示例适用于所有数据类型和目标。

REST

在使用任何请求数据之前，请先进行以下替换：

LOCATION：模型所在的区域。例如 us-central1。
PROJECT：您的项目 ID。
MODEL_ID：您的模型的 ID。
EVALUATION_ID：包含要检索的评估切片的模型评估 ID。
SLICE_ID：要获取的评估切片的 ID。
PROJECT_NUMBER：自动生成的项目编号。
EVALUATION_METRIC_SCHEMA_FILE_NAME：定义要返回的评估指标的架构文件名称，例如 classification_metrics_1.0.0。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID

如需发送请求，请选择以下方式之一：

curl

执行以下命令：

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID"

PowerShell

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

执行以下命令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID",
  "slice": {
    "dimension": "annotationSpec",
    "value": "a particular class or label"
  },
  "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/EVALUATION_METRIC_SCHEMA_FILE_NAME.yaml",
  "metrics": {
    evaluation metrics for the slice
  },
  "createTime": "2020-10-08T23:35:54.770876Z"
}

Java

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Java 设置说明执行操作。如需了解详情，请参阅 Vertex AI Java API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。


import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSliceName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    String sliceId = "YOUR_SLICE_ID";
    getModelEvaluationSliceSample(project, modelId, evaluationId, sliceId);
  }

  static void getModelEvaluationSliceSample(
      String project, String modelId, String evaluationId, String sliceId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationSliceName modelEvaluationSliceName =
          ModelEvaluationSliceName.of(project, location, modelId, evaluationId, sliceId);

      ModelEvaluationSlice modelEvaluationSlice =
          modelServiceClient.getModelEvaluationSlice(modelEvaluationSliceName);

      System.out.println("Get Model Evaluation Slice Response");
      System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
      System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
      System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
      System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

      Slice slice = modelEvaluationSlice.getSlice();
      System.out.format("Slice Dimensions: %s\n", slice.getDimension());
      System.out.format("Slice Value: %s\n", slice.getValue());
    }
  }
}

Node.js

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Node.js 设置说明执行操作。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const sliceId = 'YOUR_SLICE_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');
// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};
// Specifies the location of the api endpoint
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationSlice() {
  // Configure the parent resource
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}/slices/${sliceId}`;
  const request = {
    name,
  };

  // Get and print out a list of all the endpoints for this resource
  const [response] =
    await modelServiceClient.getModelEvaluationSlice(request);

  console.log('Get model evaluation slice');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics_Schema_Uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);

  console.log('Slice');
  const slice = response.slice;
  console.log(`\tDimension :${slice.dimension}`);
  console.log(`\tValue :${slice.value}`);
}
getModelEvaluationSlice();

Python

如需了解如何安装或更新 Python，请参阅安装 Python 版 Vertex AI SDK。如需了解详情，请参阅 Python API 参考文档。

from google.cloud import aiplatform

def get_model_evaluation_slice_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    slice_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_slice_path(
        project=project,
        location=location,
        model=model_id,
        evaluation=evaluation_id,
        slice=slice_id,
    )
    response = client.get_model_evaluation_slice(name=name)
    print("response:", response)

迭代模型

当模型未达到您的预期时，模型评估指标可为您提供调试模型的起点。例如，低精确率和召回率可能表明您的模型需要额外的训练数据或者存在不一致的标签。完美的精确率和召回率可能表明测试数据难以预测，并且可能无法有效泛化。

您可以迭代训练数据并创建新模型。创建新模型后，您可以比较现有模型和新模型的评估指标。

以下建议可帮助您改进为各项添加标签的模型（例如分类或检测模型）：

建议在您的训练数据中添加更多样本或范围更广的示例。如需更多指导，请参阅准备数据。
考虑移除没有大量样本的类或标签。样本不足会导致模型无法一致、自信地对这些类或标签进行预测。
机器无法解读类或标签的名称，也无法理解它们之间的细微差别，例如“door”和“door_with_knob”。您必须提供数据来帮助机器识别这些细微差别。
使用更多的真正例和真负例样本来扩充您的数据，尤其是靠近决策边界的样本，以降低模型混淆。
指定您自己的数据拆分（训练、验证和测试）。Vertex AI 会为每个集合随机分配项。因此，相似的重复项可能会分配在训练集和验证集中，这可能导致过拟合，进而造成测试集性能不佳。如需详细了解如何设置您自己的数据拆分，请参阅 AutoML 模型的数据拆分简介。
如果模型的评估指标包含混淆矩阵，您可以查看模型是否会混淆两个标签，其中模型预测特定标签明显比真实标签多。查看您的数据，并确保正确标记样本。
如果您的训练时间很短（较低的节点最大小时数），您可以延长训练时间来获得质量更高的模型（较高的节点数上限）。

训练模型

获取预测结果