评估 AutoML 分类和回归模型

本页面介绍了如何评估 AutoML 分类和回归模型。

Vertex AI 提供了模型评估指标，可帮助您确定模型的性能，例如精确率和召回率指标。Vertex AI 使用测试集来计算评估指标。

准备工作

在评估模型之前，必须先训练模型。

如何使用模型评估指标

模型评估指标提供模型在测试集上的表现的量化测量结果。如何解读和使用这些指标取决于您的业务需求和模型训练所要解决的问题。例如，假正例的容忍度可能低于假负例，反之亦然。这些问题会影响您重点关注的指标。

获取评估指标

您可以获取模型的一组总体评估指标，以及针对特定类或标签的评估指标。特定类或标签的评估指标也称为评估切片。以下内容介绍如何使用 Google Cloud 控制台或 API 获取总体评估指标和评估切片。

Google Cloud 控制台

在 Google Cloud 控制台的 Vertex AI 部分中，转到模型页面。

打开“模型”页面
在区域下拉列表中，选择您的模型所在的区域。
在模型列表中，点击您的模型以打开模型的评估标签页。

在评估标签页中，您可以查看模型的总体评估指标，例如平均精确率和召回率。

如果模型目标具有评估切片，则控制台会显示标签列表。您可以点击标签来查看该标签的评估指标，如以下示例所示：

API

对于每种数据类型和目标，获取评估指标的 API 请求是相同的，但输出有所不同。以下示例显示了相同的请求和不同的响应。

获取聚合模型评估指标

汇总模型评估指标提供有关整个模型的信息。如需查看有关特定切片的信息，请列出模型评估切片。

如需查看总体模型评估指标，请使用 projects.locations.models.evaluations.get 方法。

在下面选择您的目标对应的标签页：

分类

Vertex AI 会返回一组置信度指标。每个元素显示不同 confidenceThreshold 值的评估指标（从 0 开始，一直到 1）。通过查看不同的阈值，您可以看到阈值如何影响其他指标，例如精确率和召回率。

选择与您的语言或环境对应的标签页：

REST

在使用任何请求数据之前，请先进行以下替换：

LOCATION：存储模型的区域。
PROJECT：您的项目 ID。
MODEL_ID：模型资源的 ID。
PROJECT_NUMBER：自动生成的项目编号。
EVALUATION_ID：模型评估的 ID（显示在响应中）。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI，或者使用了 Cloud Shell，这会使您自动登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

执行以下命令：

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

执行以下命令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 0.97762364,
        "auRoc": 0.97566897,
        "logLoss": 0.19153881,
        "confidenceMetrics": [
          {
            "recall": 1,
            "precision": 0.5,
            "falsePositiveRate": 1,
            "f1Score": 0.6666667,
            "recallAt1": 0.90911126,
            "precisionAt1": 0.90911126,
            "falsePositiveRateAt1": 0.09088874,
            "f1ScoreAt1": 0.90911126,
            "truePositiveCount": "4467",
            "falsePositiveCount": "4467"
          },
          {
            "confidenceThreshold": 0.003269856,
            "recall": 0.9997761,
            "precision": 0.56993365,
            "falsePositiveRate": 0.7544213,
            "f1Score": 0.7260018,
            "recallAt1": 0.90911126,
            "precisionAt1": 0.90911126,
            "falsePositiveRateAt1": 0.09088874,
            "f1ScoreAt1": 0.90911126,
            "truePositiveCount": "4466",
            "falsePositiveCount": "3370",
            "falseNegativeCount": "1",
            "trueNegativeCount": "1097"
          },
          {
            "confidenceThreshold": 0.1103351,
            "recall": 0.9899261,
            "precision": 0.79819494,
            "falsePositiveRate": 0.25027984,
            "f1Score": 0.8837814,
            "recallAt1": 0.90911126,
            "precisionAt1": 0.90911126,
            "falsePositiveRateAt1": 0.09088874,
            "f1ScoreAt1": 0.90911126,
            "truePositiveCount": "4422",
            "falsePositiveCount": "1118",
            "falseNegativeCount": "45",
            "trueNegativeCount": "3349"
          },
          ...
        ],
        "confusionMatrix": {
          "annotationSpecs": [
            {
              "displayName": "1"
            },
            {
              "displayName": "2"
            }
          ],
          "rows": [
            [
              3817,
              140
            ],
            [
              266,
              244
            ]
          ]
        }
      },
      "createTime": "2020-10-09T00:19:15.463930Z",
      "sliceDimensions": [
        "annotationSpec"
      ],
      "modelExplanation": {
        "meanAttributions": [
          {
            "featureAttributions": {
              "Age": 0.022972771897912025,
              "Job": 0.031542550772428513,
              "MaritalStatus": 0.015506803058087826,
              "Education": 0.019189134240150452,
              "Default": 0.00021766019926872104,
              "Balance": 0.031217793002724648,
              "Housing": 0.06786702573299408,
              "Loan": 0.0072592208161950111,
              "Contact": 0.083566240966320038,
              "Day": 0.074894927442073822,
              "Month": 0.19679982960224152,
              "Duration": 0.35500210523605347,
              "Campaign": 0.033425047993659973,
              "PDays": 0.013902961276471615,
              "Previous": 0.0061685866676270962,
              "POutcome": 0.040467333048582077
            }
          }
        ]
      }
    }
  ]
}

Java

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Java 设置说明执行操作。如需了解详情，请参阅 Vertex AI Java API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。


import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationTabularClassificationSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    getModelEvaluationTabularClassification(project, modelId, evaluationId);
  }

  static void getModelEvaluationTabularClassification(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);
      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Tabular Classification Response");
      System.out.format("\tName: %s\n", modelEvaluation.getName());
      System.out.format("\tMetrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("\tMetrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("\tCreate Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("\tSlice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Node.js 设置说明执行操作。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationTabularClassification() {
  // Configure the parent resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation tabular classification response');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);
  console.log(`\tSlice dimensions : ${response.sliceDimensions}`);

  const modelExplanation = response.modelExplanation;
  console.log('\tModel explanation');
  if (!modelExplanation) {
    console.log('\t\t{}');
  } else {
    const meanAttributions = modelExplanation.meanAttributions;
    if (!meanAttributions) {
      console.log('\t\t\t []');
    } else {
      for (const meanAttribution of meanAttributions) {
        console.log('\t\tMean attribution');
        console.log(
          `\t\t\tBaseline output value : \
            ${meanAttribution.baselineOutputValue}`
        );
        console.log(
          `\t\t\tInstance output value : \
            ${meanAttribution.instanceOutputValue}`
        );
        console.log(
          `\t\t\tFeature attributions : \
            ${JSON.stringify(meanAttribution.featureAttributions)}`
        );
        console.log(`\t\t\tOutput index : ${meanAttribution.outputIndex}`);
        console.log(
          `\t\t\tOutput display name : \
            ${meanAttribution.outputDisplayName}`
        );
        console.log(
          `\t\t\tApproximation error : \
            ${meanAttribution.approximationError}`
        );
      }
    }
  }
}
getModelEvaluationTabularClassification();

Python

如需了解如何安装或更新 Vertex AI SDK for Python，请参阅安装 Vertex AI SDK for Python。如需了解详情，请参阅 Python API 参考文档。

from google.cloud import aiplatform


def get_model_evaluation_tabular_classification_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

回归

选择与您的语言或环境对应的标签页：

REST

在使用任何请求数据之前，请先进行以下替换：

LOCATION：存储模型的区域。
PROJECT：您的项目 ID。
MODEL_ID：模型资源的 ID。
PROJECT_NUMBER：自动生成的项目编号。
EVALUATION_ID：模型评估的 ID（显示在响应中）。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如需发送请求，请选择以下方式之一：

curl

执行以下命令：

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

执行以下命令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/regression_metrics_1.0.0.yaml",
      "metrics": {
        "rootMeanSquaredError": 2553.6309,
        "meanAbsoluteError": 1373.3932,
        "meanAbsolutePercentageError": "Infinity",
        "rSquared": 0.060764354,
        "rootMeanSquaredLogError": "NaN"
      },
      "createTime": "2020-10-09T01:20:37.045482Z",
      "modelExplanation": {
        "meanAttributions": [
          {
            "featureAttributions": {
              "Age": 0.22535169124603271,
              "Job": 0.049311652779579163,
              "MaritalStatus": 0.033439181745052338,
              "Education": 0.10934026539325714,
              "Default": 0.021301545202732086,
              "Housing": 0.0631907731294632,
              "Loan": 0.055760543793439865,
              "Contact": 0.010930608958005905,
              "Day": 0.14066702127456665,
              "Month": 0.17570944130420685,
              "Duration": 0.054339192807674408,
              "Campaign": 0.015468073077499866,
              "PDays": 0.020416950806975365,
              "Previous": 0.0037290120963007212,
              "POutcome": 0.0040646209381520748,
              "Deposit": 0.016979435458779335
            }
          }
        ]
      }
    }
  ]
}

Java

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Java 设置说明执行操作。如需了解详情，请参阅 Vertex AI Java API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。


import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationTabularRegressionSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    getModelEvaluationTabularRegression(project, modelId, evaluationId);
  }

  static void getModelEvaluationTabularRegression(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);
      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Tabular Regression Response");
      System.out.format("\tName: %s\n", modelEvaluation.getName());
      System.out.format("\tMetrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("\tMetrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("\tCreate Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("\tSlice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Node.js 设置说明执行操作。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationTabularRegression() {
  // Configure the parent resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation tabular regression response');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);
  console.log(`\tSlice dimensions : ${response.sliceDimensions}`);

  const modelExplanation = response.modelExplanation;
  console.log('\tModel explanation');
  if (!modelExplanation) {
    console.log('\t\t{}');
  } else {
    const meanAttributions = modelExplanation.meanAttributions;
    if (!meanAttributions) {
      console.log('\t\t\t []');
    } else {
      for (const meanAttribution of meanAttributions) {
        console.log('\t\tMean attribution');
        console.log(
          `\t\t\tBaseline output value : \
            ${meanAttribution.baselineOutputValue}`
        );
        console.log(
          `\t\t\tInstance output value : \
            ${meanAttribution.instanceOutputValue}`
        );
        console.log(
          `\t\t\tFeature attributions : \
            ${JSON.stringify(meanAttribution.featureAttributions)}`
        );
        console.log(`\t\t\tOutput index : ${meanAttribution.outputIndex}`);
        console.log(
          `\t\t\tOutput display name : \
            ${meanAttribution.outputDisplayName}`
        );
        console.log(
          `\t\t\tApproximation error : \
            ${meanAttribution.approximationError}`
        );
      }
    }
  }
}
getModelEvaluationTabularRegression();

Python

如需了解如何安装或更新 Vertex AI SDK for Python，请参阅安装 Vertex AI SDK for Python。如需了解详情，请参阅 Python API 参考文档。

from google.cloud import aiplatform


def get_model_evaluation_tabular_regression_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

列出所有评估切片（仅限分类模型）

projects.locations.models.evaluations.slices.list 方法可列出您的模型的所有评估切片。您必须有模型的评估 ID，此 ID 可以在查看汇总评估指标时获得。

您可以使用模型评估切片来确定模型在特定标签上的表现。value 字段指示与指标对应的标签。

REST

在使用任何请求数据之前，请先进行以下替换：

LOCATION：模型所在的区域。例如 us-central1。
PROJECT：您的项目 ID。
MODEL_ID：您的模型的 ID。
EVALUATION_ID：包含要列出评估切片的模型评估的 ID。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices

如需发送请求，请选择以下方式之一：

curl

执行以下命令：

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices"

PowerShell

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

执行以下命令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "modelEvaluationSlices": [
    {
      "name": "projects/693884908213/locations/us-central1/models/705305922892726272/evaluations/4515484958386859492/slices/1785244630562158241",
      "slice": {
        "dimension": "annotationSpec",
        "value": "2"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 0.6108714,
        "auRoc": 0.9362428,
        "logLoss": 0.9680687,
        "confidenceMetrics": [
          {
            "recall": 1,
            "precision": 0.11417058,
            "falsePositiveRate": 1,
            "f1Score": 0.20494273,
            "recallAt1": 0.47843137,
            "precisionAt1": 0.6354167,
            "falsePositiveRateAt1": 0.035380337,
            "f1ScoreAt1": 0.5458613,
            "truePositiveCount": "510",
            "falsePositiveCount": "3957"
          },
          {
            "confidenceThreshold": 0.003269856,
            "recall": 0.9980392,
            "precision": 0.15108341,
            "falsePositiveRate": 0.7227698,
            "f1Score": 0.26243877,
            "recallAt1": 0.47843137,
            "precisionAt1": 0.6354167,
            "falsePositiveRateAt1": 0.035380337,
            "f1ScoreAt1": 0.5458613,
            "truePositiveCount": "509",
            "falsePositiveCount": "2860",
            "falseNegativeCount": "1",
            "trueNegativeCount": "1097"
          },
          {
            "confidenceThreshold": 0.016592776,
            "recall": 0.9882353,
            "precision": 0.23344141,
            "falsePositiveRate": 0.41824615,
            "f1Score": 0.37766954,
            "recallAt1": 0.47843137,
            "precisionAt1": 0.6354167,
            "falsePositiveRateAt1": 0.035380337,
            "f1ScoreAt1": 0.5458613,
            "truePositiveCount": "504",
            "falsePositiveCount": "1655",
            "falseNegativeCount": "6",
            "trueNegativeCount": "2302"
          },
          ...
        ]
      },
      "createTime": "2020-10-09T00:19:15.480435Z"
    },
    {
      "name": "projects/693884908213/locations/us-central1/models/705305922892726272/evaluations/4515484958386859492/slices/8107013027312442123",
      "slice": {
        "dimension": "annotationSpec",
        "value": "1"
      },
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
      "metrics": {
        "auPrc": 0.9916441,
        "auRoc": 0.93830043,
        "logLoss": 0.09145534,
        "confidenceMetrics": [
          {
            "recall": 1,
            "precision": 0.8858294,
            "falsePositiveRate": 1,
            "f1Score": 0.93945867,
            "recallAt1": 0.96461964,
            "precisionAt1": 0.9348518,
            "falsePositiveRateAt1": 0.52156866,
            "f1ScoreAt1": 0.94950247,
            "truePositiveCount": "3957",
            "falsePositiveCount": "510"
          },
          {
            "confidenceThreshold": 0.064618945,
            "recall": 0.9997473,
            "precision": 0.88639927,
            "falsePositiveRate": 0.9941176,
            "f1Score": 0.93966746,
            "recallAt1": 0.96461964,
            "precisionAt1": 0.9348518,
            "falsePositiveRateAt1": 0.52156866,
            "f1ScoreAt1": 0.94950247,
            "truePositiveCount": "3956",
            "falsePositiveCount": "507",
            "falseNegativeCount": "1",
            "trueNegativeCount": "3"
          },
          ...

Java

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Java 设置说明执行操作。如需了解详情，请参阅 Vertex AI Java API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。


import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class ListModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    listModelEvaluationSliceSample(project, modelId, evaluationId);
  }

  static void listModelEvaluationSliceSample(String project, String modelId, String evaluationId)
      throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      for (ModelEvaluationSlice modelEvaluationSlice :
          modelServiceClient.listModelEvaluationSlices(modelEvaluationName).iterateAll()) {
        System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
        System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
        System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
        System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

        Slice slice = modelEvaluationSlice.getSlice();
        System.out.format("Slice Dimensions: %s\n", slice.getDimension());
        System.out.format("Slice Value: %s\n\n", slice.getValue());
      }
    }
  }
}

Node.js

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Node.js 设置说明执行操作。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function listModelEvaluationSlices() {
  // Configure the parent resources
  const parent = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    parent,
  };

  // Get and print out a list of all the evaluation slices for this resource
  const [response] =
    await modelServiceClient.listModelEvaluationSlices(request);
  console.log('List model evaluation response', response);
  console.log(response);
}
listModelEvaluationSlices();

Python

如需了解如何安装或更新 Vertex AI SDK for Python，请参阅安装 Vertex AI SDK for Python。如需了解详情，请参阅 Python API 参考文档。

from google.cloud import aiplatform


def list_model_evaluation_slices_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    parent = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.list_model_evaluation_slices(parent=parent)
    for model_evaluation_slice in response:
        print("model_evaluation_slice:", model_evaluation_slice)

获取单个切片的指标

如需查看单个切片的评估指标，请使用 projects.locations.models.evaluations.slices.get 方法。您必须拥有切片 ID（该 ID 会在您列出所有切片时提供）。以下示例适用于所有数据类型和目标。

REST

在使用任何请求数据之前，请先进行以下替换：

LOCATION：模型所在的区域。例如 us-central1。
PROJECT：您的项目 ID。
MODEL_ID：您的模型的 ID。
EVALUATION_ID：包含要检索的评估切片的模型评估 ID。
SLICE_ID：要获取的评估切片的 ID。
PROJECT_NUMBER：自动生成的项目编号。
EVALUATION_METRIC_SCHEMA_FILE_NAME：定义要返回的评估指标的架构文件名称，例如 classification_metrics_1.0.0。

HTTP 方法和网址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID

如需发送请求，请选择以下方式之一：

curl

执行以下命令：

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID"

PowerShell

注意：以下命令假定您已使用您的用户账号通过运行 gcloud init 或 gcloud auth login 登录 gcloud CLI。您可以运行 gcloud auth list 来检查当前活跃的账号。

执行以下命令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID" | Select-Object -Expand Content

您应该收到类似以下内容的 JSON 响应：

响应

{
  "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID",
  "slice": {
    "dimension": "annotationSpec",
    "value": "a particular class or label"
  },
  "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/EVALUATION_METRIC_SCHEMA_FILE_NAME.yaml",
  "metrics": {
    evaluation metrics for the slice
  },
  "createTime": "2020-10-08T23:35:54.770876Z"
}

Java

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Java 设置说明执行操作。如需了解详情，请参阅 Vertex AI Java API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。


import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSliceName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    String sliceId = "YOUR_SLICE_ID";
    getModelEvaluationSliceSample(project, modelId, evaluationId, sliceId);
  }

  static void getModelEvaluationSliceSample(
      String project, String modelId, String evaluationId, String sliceId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationSliceName modelEvaluationSliceName =
          ModelEvaluationSliceName.of(project, location, modelId, evaluationId, sliceId);

      ModelEvaluationSlice modelEvaluationSlice =
          modelServiceClient.getModelEvaluationSlice(modelEvaluationSliceName);

      System.out.println("Get Model Evaluation Slice Response");
      System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
      System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
      System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
      System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

      Slice slice = modelEvaluationSlice.getSlice();
      System.out.format("Slice Dimensions: %s\n", slice.getDimension());
      System.out.format("Slice Value: %s\n", slice.getValue());
    }
  }
}

Node.js

在尝试此示例之前，请按照《Vertex AI 快速入门：使用客户端库》中的 Node.js 设置说明执行操作。如需了解详情，请参阅 Vertex AI Node.js API 参考文档。

如需向 Vertex AI 进行身份验证，请设置应用默认凭据。如需了解详情，请参阅为本地开发环境设置身份验证。

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const sliceId = 'YOUR_SLICE_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');
// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};
// Specifies the location of the api endpoint
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationSlice() {
  // Configure the parent resource
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}/slices/${sliceId}`;
  const request = {
    name,
  };

  // Get and print out a list of all the endpoints for this resource
  const [response] =
    await modelServiceClient.getModelEvaluationSlice(request);

  console.log('Get model evaluation slice');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics_Schema_Uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);

  console.log('Slice');
  const slice = response.slice;
  console.log(`\tDimension :${slice.dimension}`);
  console.log(`\tValue :${slice.value}`);
}
getModelEvaluationSlice();

Python

如需了解如何安装或更新 Vertex AI SDK for Python，请参阅安装 Vertex AI SDK for Python。如需了解详情，请参阅 Python API 参考文档。

from google.cloud import aiplatform


def get_model_evaluation_slice_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    slice_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_slice_path(
        project=project,
        location=location,
        model=model_id,
        evaluation=evaluation_id,
        slice=slice_id,
    )
    response = client.get_model_evaluation_slice(name=name)
    print("response:", response)

模型评估指标

Vertex AI 会返回几个不同的评估指标，例如精确率、召回率和置信度阈值。Vertex AI 返回的指标取决于您的模型目标。例如，与图像对象检测模型相比，Vertex AI 为图像分类模型提供了不同的评估指标。

架构文件决定了 Vertex AI 为每个目标提供的评估指标。

您可以从以下 Cloud Storage 位置查看和下载架构文件：
gs://google-cloud-aiplatform/schema/modelevaluation/

评估指标包括：

回归

MAE：平均绝对误差 (MAE) 指的是目标值与预测值之间的平均绝对差。此指标的范围在零到无穷大之间；值越小表示模型质量越高。
RMSE：均方根误差是目标值与预测值之间的平均平方差的平方根。RMSE 对离群值比 MAE 更敏感，因此如果您担心大的误差，那么 RMSE 可能是一个更有用的评估指标。与 MAE 类似，值越小表示模型质量越高（0 表示完美的预测模型）。
RMSLE：均方根对数误差指标与 RMSE 类似，不同的是它使用预测值和实际值加 1 的自然对数。RMSLE 对预测不足的罚分比过度预测更重。如果您不希望对大预测值误差的罚分比对小预测值的更重，则此指标也非常适合您的需求。此指标的范围在零到无穷大之间；值越小表示模型质量越高。只有当所有标签值和预测值均为非负值时，才会返回 RMSLE 评估指标。
r^2：r 平方 (r^2) 是标签值与预测值之间的皮尔逊相关系数的平方，该指标的范围介于 0 到 1 之间。值越大，表示越接近回归线。
MAPE：平均绝对百分比误差 (MAPE) 指的是标签与预测值之间的平均绝对百分比差。该指标的范围介于 0 到无穷大之间；值越低表示模型质量越高。
如果目标列包含任何为 0 的值，则 MAPE 不会显示。在这种情况下，MAPE 未经定义。
模型特征归因：Vertex AI 显示每个特征对模型的影响程度。这些值是以每个特征的百分比形式提供的：百分比越高，特征对模型训练的影响就越大。应查看此信息，以确保所有最重要的特征对您的数据和业务问题都有所帮助。如需了解详情，请参阅用于分类和回归的特征归因

后续步骤

准备好通过分类或回归模型进行预测后，您可以采用以下两种方法开始预测：

此外，您可以：

查看模型的架构。
了解如何导出模型。