View pipeline job logs

Stay organized with collections Save and categorize content based on your preferences.

After you define and build a pipeline, you can use Cloud Logging to create log entries to help you monitor events such as pipeline failures. You can create custom log-based metrics that send notifications when the rate of pipeline failures reaches a given threshold.

This page describes how to use Cloud Logging in the Google Cloud console and gcloud CLI to view and query log entries for your Vertex AI Pipelines job.

This feature has costs associated with it. See Cloud Logging pricing for more information.

View Vertex AI logs using the Google Cloud console and gcloud CLI

Vertex AI Pipelines uses Cloud Logging, monitored resources, and payloads for Vertex AI.

To learn more about the log entries for Vertex AI, see Google Cloud platform logs.

To view logs for your Vertex AI Pipelines jobs:

Console

  1. Enable the Cloud Logging API:

    Enable the API

  2. In the Google Cloud console, go to the Logs explorer:

    Go to Logs explorer

  3. Select an existing Vertex AI project at the top of the page.

  4. In the Query builder, add the following:

    • Resource: Select Vertex Pipelines Job. In the dialog, select a Vertex AI Pipelines job.
    • Log names: Scroll to the Vertex AI section and select aiplatform.googlapis.com/pipeline_job_events.
    • Severity: Select a log level.
    • Time range: Select a preset range or create a custom range.

gcloud

  1. Run the following command to enable the Cloud Logging API:

    gcloud services enable logging.googleapis.com
    
  2. Execute the gcloud logging read command:

    Linux, macOS, or Cloud Shell

    gcloud logging read "projects/PROJECT_ID/logs/aiplatform.googleapis.com/pipeline_job_events" \
        --limit=LIMIT
    

    Windows (PowerShell)

    gcloud logging read "projects/PROJECT_ID/logs/aiplatform.googleapis.com/pipeline_job_events" `
        --limit=LIMIT
    

    Windows (cmd.exe)

    gcloud logging read "projects/PROJECT_ID/logs/aiplatform.googleapis.com/pipeline_job_events" ^
        --limit=LIMIT
    
     

Build custom metrics in the Logs Explorer

You can use the Logs Explorer in the Google Cloud console to create custom log-based metrics that track and analyze patterns within your pipeline logs.

Following are examples of some custom metrics you can create.

Create a custom metric for final pipeline state

To create a custom log-based counter metric that monitors the final state of your pipeline jobs, do the following:

  1. Navigate to the Logs Explorer:

    Go to the Logs Explorer

  2. Select the appropriate Cloud project.

  3. In the Resource drop-down menu, select Vertex Pipelines Job.

  4. In the Location drop-down menu, select your pipeline's location.

  5. In the All pipeline_job_id drop-down menu, select the ID of your pipeline job.

  6. Click Apply.

  7. Click Create Metric.

  8. In the Create logs metric screen:

    1. Set the Metric type: Select Counter.

    2. Set the following fields in the Details section:

      • Log metric name: Enter a name for your log metric, for example, Pipeline_state_final. Choose a name that is unique among the logs-based metrics in your Cloud project. Some naming restrictions apply; see Troubleshooting for details.
      • Description: Enter a description for the metric.
      • Units: Leave this field blank or enter the digit 1.
    3. Define your metric filter in the Filter selection section. Create a filter that collects only the log entries that you want to count in your metric using the logging query language, for example:

      resource.type="aiplatform.googleapis.com/PipelineJob"
      resource.labels.location="us-central1"
      jsonPayload.state="PIPELINE_STATE_SUCCEEDED" OR
      "PIPELINE_STATE_FAILED" OR "PIPELINE_STATE_CANCELLED"
      

      You can also use regular expressions to create your metric's filters.

      Click Preview logs to open a panel showing you the log entries that match your filter.

    4. (Optional) Add a label in the Labels section. For instructions on creating labels, see Create a label.

    5. Click Create metric to create the metric.

Create a custom metric for failed pipeline jobs

To create a custom log-based counter metric that monitors failed pipeline jobs, do the following:

  1. Navigate to the Logs Explorer:

    Go to the Logs Explorer

  2. Select the appropriate Cloud project.

  3. In the Resource drop-down menu, select Vertex Pipelines Job.

  4. In the Location drop-down menu, select your pipeline's location.

  5. In the All pipeline_job_id drop-down menu, select the ID of your pipeline job.

  6. Click Apply.

  7. Click Create Metric.

  8. In the Create logs metric screen:

    1. Set the Metric type: Select Counter.

    2. Set the following fields in the Details section:

      • Log metric name: Enter a name for your log metric, for example, pipelinejob_failed. Choose a name that is unique among the logs-based metrics in your Cloud project. Some naming restrictions apply; see Troubleshooting for details.
      • Description: Enter a description for the metric.
      • Units: Leave this field blank or enter the digit 1.
    3. Define your metric filter in the Filter selection section. Create a filter that collects only the log entries that you want to count in your metric using the logging query language, for example:

      resource.type="aiplatform.googleapis.com/PipelineJob"
      resource.labels.location="us-central1"
      jsonPayload.state="PIPELINE_STATE_FAILED"
      

      You can also use regular expressions to create your metric's filters.

      Click Preview logs to open a panel showing you the log entries that match your filter.

    4. (Optional) Add a label in the Labels section. For instructions on creating labels, see Create a label.

    5. Click Create metric to create the metric.

View pipeline job failure rate

The pipeline job failure rate is calculated as the ratio of the number of pipeline jobs in the final state to the number of failed pipeline jobs. To create a dashboard to monitor the pipeline job failure rate, do the following:

  1. Create a metric for monitoring the pipeline jobs in the final state. For more information, see Create a custom metric for final pipeline state.

  2. Create a metric for monitoring the pipeline jobs in the failed state. For more information, see Create a custom metric for failed pipeline jobs.

  3. In the Monitoring section of the Google Cloud console, go to the Metrics Explorer page.

    Go to the Metrics Explorer page

  4. In the Configuration tab, do the following:

    1. Click Add another metric.

    2. Select the Display metrics as ratio option. When you select this option:

      • The Time series A pane is renamed Numerator.

      • The Time series B pane is renamed Denominator.

    3. In the Numerator pane, click Select a metric to select the metric created for failed pipeline jobs.

    4. In the Denominator pane, click Select a metric to select the metric created for final state pipeline jobs.

    5. (Optional) Configure the numerator and denominator by adding filters, or by updating the grouping fields and alignment parameters.

      For more information about how to add filters, or update the grouping fields and alignment parameters, see Chart a ratio of metrics.

    After you select the numerator and the denominator, the pipeline job failure rate is displayed in the graph.

    • After you generate the pipeline job failure rate graph, you can add it to your custom dashboard. For more information, see Save a chart for future reference.

    • To copy the URL containing the graph configuration, click the ellipsis icon on the top right corner of the graph, and then click Share by URL.