Vertex AI Model Monitoring for batch predictions

Stay organized with collections Save and categorize content based on your preferences.

This page describes how to configure batch prediction job requests to include one-time Model Monitoring analysis. For batch predictions, Model Monitoring supports feature skew detection for categorical and numerical input features.

To create a batch prediction job with Model Monitoring skew analysis, you must include both your batch prediction input data and original training data for your model in the request. You can only add Model Monitoring analysis when creating new batch prediction jobs.

For more information about skew, see Introduction to Model Monitoring.

For instructions on how to set up Model Monitoring for online (real-time) predictions, see Using Model Monitoring.

Prerequisites

To use Model Monitoring with batch predictions, complete the following:

  1. Have an available model in Vertex AI Model Registry that is either a tabular AutoML or tabular custom training type.

  2. Upload your training data to Cloud Storage or BigQuery and obtain the URI link to the data.

    • For models trained with AutoML, you can use the dataset id for your training dataset instead.
  3. Model Monitoring compares the training data to the batch prediction output. Make sure you are using supported file formats for the training data and batch prediction output:

    Model type Training data Batch prediction output
    Custom-trained CSV, JSONL, BigQuery, TfRecord(tf.train.Example) JSONL
    AutoML tabular CSV, JSONL, BigQuery, TfRecord(tf.train.Example) CSV, JSONL, BigQuery, TfRecord(Protobuf.Value)

Request a batch prediction

You can use the following methods to add Model Monitoring configurations to batch prediction jobs:

Console

Follow the instructions to make a batch prediction request with Model Monitoring enabled:

REST API

Follow the instructions to make a batch prediction request using the REST API:

When you create the batch prediction request, add the following Model Monitoring configuration to the request JSON body:

"modelMonitoringConfig": {
 "alertConfig": {
   "emailAlertConfig": {
     "userEmails": "EMAIL_ADDRESS"
   }
 },
 "objectiveConfigs": [
   {
     "trainingDataset": {
       "dataFormat": "csv",
       "gcsSource": {
         "uris": [
           "TRAINING_DATASET"
         ]
       }
     },
     "trainingPredictionSkewDetectionConfig": {
       "skewThresholds": {
         "FEATURE_1": {
           "value": VALUE_1
         },
         "FEATURE_2": {
           "value": VALUE_2
         }
       }
     }
   }
 ]
}

where:

  • EMAIL_ADDRESS is the email address where you want to receive alerts from Model Monitoring. For example, example@example.com.

  • TRAINING_DATASET is the link to the training dataset stored in Cloud Storage.

    • To use a link to a BigQuery training dataset, replace the the gcsSource field with the following:
    "bigquerySource": {
        {
          "inputUri": "TRAINING_DATASET"
        }
     }
    
    • To use a link to an AutoML model, replace the gcsSource field with the following:
    "dataset": "TRAINING_DATASET"
  • FEATURE_1:VALUE_1 and FEATURE_2:VALUE_2 is the alerting threshold for each feature you want to monitor. For example, if you specify Age=0.4, Model Monitoring logs an alert when the statistical distance between the input and baseline distributions for the Age feature exceeds 0.4. By default, every categorical and numerical feature is monitored with threshold values of 0.3.

For more information about Model Monitoring configurations, see the Monitoring job reference.

Python

See the example notebook to run a batch prediction job with Model Monitoring for a custom tabular model.

Model Monitoring automatically notifies you of job updates and alerts through email.

Analyze skew data

You can use Google Cloud console to visualize the distributions of each monitored feature and learn which changes led to skew over time:

  1. To navigate to the feature distribution histograms in the Google Cloud console, go to the Batch predictions page.

    Go to Batch predictions

  2. On the Batch predictions page, click the batch prediction job you want to analyze.

  3. Click the Model Monitoring Alerts tab to view a list of the model's input features, along with pertinent information, such as the alert threshold for each feature.

  4. To analyze a feature, click the name of the feature. A page shows the feature distribution histograms for that feature.

    Visualizing data distribution as histograms lets you quickly focus on the changes that occurred in the data. Afterward, you might decide to adjust your feature generation pipeline or retrain the model.

    Histograms showing example input data distribution and training
            data distribution for skew detection.

What's next