Evaluate AutoML forecast models

This page shows you how to evaluate your AutoML forecast models using model evaluation metrics. These metrics provide quantitative measurements of how your model performed on the test set. How you interpret and use those metrics depends on your business need and the problem your model is trained to solve. For example, you might have a lower tolerance for false positives than for false negatives or the other way around. These kinds of questions affect which metrics you would focus on.

Before you begin

Before you can evaluate a model, you must train it and wait for the training to complete.

Use the console or the API to check the status of your training job.

Google Cloud console

1. In the Google Cloud console, in the Vertex AI section, go to the Training page.

   Go to the Training page

2. If the status of your training job is "Training", you must continue to wait for the training job to finish. If the status of your training job is "Finished", you are ready to begin model evaluation.

API

Select a tab that corresponds to your language or environment:

REST & CMD LINE

Before using any of the request data, make the following replacements:

• LOCATION: Region where your model is stored.
• PROJECT: Your project ID.
• TRAINING_PIPELINE_ID: ID of the training pipeline.

HTTP method and URL:

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/trainingPipelines/TRAINING_PIPELINE_ID

To send your request, choose one of these options:

curl

Note: Ensure you have set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path for your service account private key file.

Execute the following command:

curl -X GET \
    -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
    "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/trainingPipelines/TRAINING_PIPELINE_ID"

PowerShell

Note: Ensure you have set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path for your service account private key file.

Execute the following command:

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/trainingPipelines/TRAINING_PIPELINE_ID" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

Response

  If the model is still being trained:

  {
    "name": "projects/PROJECT_NUMBER/locations/LOCATION/trainingPipelines/TRAINING_PIPELINE_ID",
    ...
    "modelToUpload": {
      "displayName": "MODEL_DISPLAY_NAME"
    },
    "state": "PIPELINE_STATE_RUNNING",
    ...
  }

  If the model training is complete:

  {
    "name": "projects/PROJECT_NUMBER/locations/LOCATION/trainingPipelines/TRAINING_PIPELINE_ID",
    ...
    "modelToUpload": {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID",
      "displayName": "MODEL_DISPLAY_NAME",
      "versionID": "1"
    },
    "state": "PIPELINE_STATE_SUCCEEDED",
    ...
  }

Getting evaluation metrics

You can get an aggregate set of evaluation metrics for your model. The following content describes how to get these metrics using the Google Cloud console or API.

Google Cloud console

1. In the Google Cloud console, in the Vertex AI section, go to the Models page.

   Go to the Models page

2. In the Region drop-down, select the region where your model is located.

3. From the list of models, select your model.

4. Select your model's version number.

5. In the Evaluate tab, you can view your model's aggregate evaluation metrics.

API

To view aggregate model evaluation metrics, use the projects.locations.models.evaluations.get method.

Select a tab that corresponds to your language or environment:

REST & CMD LINE

Before using any of the request data, make the following replacements:

• LOCATION: Region where your model is stored.
• PROJECT: Your project ID.
• MODEL_ID: The ID of the model resource. The MODEL_ID appears in the training pipeline after model training is successfully completed. Refer to the Before you begin section to get the MODEL_ID.

HTTP method and URL:

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

To send your request, choose one of these options:

curl

Note: Ensure you have set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path for your service account private key file.

Execute the following command:

curl -X GET \
    -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
    "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

Note: Ensure you have set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path for your service account private key file.

Execute the following command:

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

Response

{
  "modelEvaluations": [
    {
      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",
      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/forecasting_metrics_1.0.0.yaml",
      "metrics": {
        "rootMeanSquaredError": 719.0045,
        "meanAbsoluteError": 487.0792,
        "meanAbsolutePercentageError": "Infinity",
        "rSquared": 0.8837288,
        "rootMeanSquaredLogError": "NaN",
        "quantileMetrics": [
          {
            "quantile": 0.2,
            "scaledPinballLoss": 157.34422,
            "observedQuantile": 0.15918367346938775
          },
          {
            "quantile": 0.5,
            "scaledPinballLoss": 243.5396,
            "observedQuantile": 0.45551020408163267
          },
          {
            "quantile": 0.8,
            "scaledPinballLoss": 175.39418,
            "observedQuantile": 0.81183673469387752
          }
        ]
      },
      "createTime": "2021-04-13T01:00:54.091953Z"
    }
  ]
}

Model evaluation metrics

A schema file determines which evaluation metrics Vertex AI provides for each objective.

You can view and download schema files from the following Cloud Storage location:
gs://google-cloud-aiplatform/schema/modelevaluation/

The evaluation metrics for forecasting models are:

• MAE: The mean absolute error (MAE) is the average absolute difference between the target values and the predicted values. This metric ranges from zero to infinity; a lower value indicates a higher quality model.
• MAPE: Mean absolute percentage error (MAPE) is the average absolute percentage difference between the labels and the predicted values. This metric ranges between zero and infinity; a lower value indicates a higher quality model.
  MAPE is not shown if the target column contains any 0 values. In this case, MAPE is undefined.
• RMSE: The root-mean-squared error is the square root of the average squared difference between the target and predicted values. RMSE is more sensitive to outliers than MAE,so if you're concerned about large errors, then RMSE can be a more useful metric to evaluate. Similar to MAE, a smaller value indicates a higher quality model (0 represents a perfect predictor).
• RMSLE: The root-mean-squared logarithmic error metric is similar to RMSE, except that it uses the natural logarithm of the predicted and actual values plus 1. RMSLE penalizes under-prediction more heavily than over-prediction. It can also be a good metric when you don't want to penalize differences for large prediction values more heavily than for small prediction values. This metric ranges from zero to infinity; a lower value indicates a higher quality model. The RMSLE evaluation metric is returned only if all label and predicted values are non-negative.
• r^2: r squared (r^2) is the square of the Pearson correlation coefficient between the labels and predicted values. This metric ranges between zero and one; a higher value indicates a higher quality model.
• Quantile: The percent quantile, which indicates the probability that an observed value will be below the predicted value. For example, at the 0.2 quantile, the observed values are expected to be lower than the predicted values 20% of the time. Vertex AI provides this metric if you specify minimize-quantile-loss for the optimization objective.
• Observed quantile: Shows the percentage of true values that were less than the predicted value for a given quantile. Vertex AI provides this metric if you specify minimize-quantile-loss for the optimization objective.
• Scaled pinball loss: The scaled pinball loss at a particular quantile. A lower value indicates a higher quality model at the given quantile. Vertex AI provides this metric if you specify minimize-quantile-loss for the optimization objective.
• Model feature attributions: Vertex AI shows you how much each feature impacts a model. The values are provided as a percentage for each feature: the higher the percentage, the more impact the feature had on model training. Review this information to ensure that all of the most important features make sense for your data and business problem. For more information, see Vertex Explainable AI.

What's next

• Get predictions from your forecast model.
• Learn how to export your model.