View pipeline job logs

After you define, build, and run a pipeline, you can use Cloud Logging to create log entries to help you monitor events such as pipeline failures. With Cloud Logging, you can create custom log-based metrics and alerts. For example, you might want to receive a notification when the rate of a pipeline exceeds a given threshold.

This page describes how to use Cloud Logging in the Google Cloud console and Google Cloud CLI to view and query log entries for your Vertex AI Pipelines job and pipeline tasks. This page also describes how to route logs to a Cloud Pub/Sub sink.

This feature has costs associated with it. For more information, see Cloud Logging pricing.

View Vertex AI logs

You can view logs for your Vertex AI Pipelines jobs and tasks.

View Vertex AI logs for pipeline jobs

To view logs for your Vertex AI Pipelines jobs:

Console

1. Enable the Cloud Logging API:

   Enable the API

2. In the Google Cloud console, go to the Logs Explorer:

   Go to the Logs Explorer

3. Select an existing Vertex AI project at the top of the page.

4. In the Query builder, add the following:

   • Resource: Select Vertex Pipelines Job. In the dialog, select a Vertex AI Pipelines job.
   • Log names: Scroll to the Vertex AI section and select aiplatform.googlapis.com/pipeline_job_events.
   • Severity: Select a log level.
   • Time range: Select a preset range or create a custom range.

gcloud

1. Run the following command to enable the Cloud Logging API:

   gcloud services enable logging.googleapis.com
   

2. Execute the gcloud logging read command:

   Linux, macOS, or Cloud Shell

   Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

   gcloud logging read "projects/PROJECT_ID/logs/aiplatform.googleapis.com/pipeline_job_events" \
       --limit=LIMIT
   

   Windows (PowerShell)

   Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

   gcloud logging read "projects/PROJECT_ID/logs/aiplatform.googleapis.com/pipeline_job_events" `
       --limit=LIMIT
   

   Windows (cmd.exe)

   Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

   gcloud logging read "projects/PROJECT_ID/logs/aiplatform.googleapis.com/pipeline_job_events" ^
       --limit=LIMIT
   

    

Learn more about log entries for the Vertex AI API.

View Vertex AI logs for pipeline tasks

To view logs for your Vertex AI Pipelines tasks:

Console

1. Enable the Cloud Logging API:

   Enable the API

2. In the Google Cloud console, go to the Logs Explorer:

   Go to the Logs Explorer

3. Select an existing Vertex AI project at the top of the page.

4. In the Query builder, add the following:

   • Resource: Select Vertex Pipelines Job. In the dialog, select a Vertex AI Pipelines job.
   • Log names: Scroll to the Vertex AI section and select aiplatform.googlapis.com/pipeline_job_task_events.
   • Severity: Select a log level.
   • Time range: Select a preset range or create a custom range.

gcloud

1. Run the following command to enable the Cloud Logging API:

   gcloud services enable logging.googleapis.com
   

2. Execute the gcloud logging read command:

   Linux, macOS, or Cloud Shell

   Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

   gcloud logging read "projects/PROJECT_ID/logs/aiplatform.googleapis.com/pipeline_job_task_events" \
       --limit=LIMIT
   

   Windows (PowerShell)

   Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

   gcloud logging read "projects/PROJECT_ID/logs/aiplatform.googleapis.com/pipeline_job_task_events" `
       --limit=LIMIT
   

   Windows (cmd.exe)

   Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

   gcloud logging read "projects/PROJECT_ID/logs/aiplatform.googleapis.com/pipeline_job_task_events" ^
       --limit=LIMIT
   

    

Learn more about log entries for the Vertex AI API.

Build custom metrics in the Logs Explorer

You can use the Logs Explorer in the Google Cloud console to create custom log-based metrics that track and analyze patterns within your pipeline logs.

Examples of custom metrics

This section illustrates examples of custom metrics you can create. These include the following:

• Create a custom metric for failed pipeline jobs

• Create a custom metric for final pipeline state

• View pipeline job failure rate

Create a custom metric for failed pipeline jobs

To create a custom log-based counter metric that monitors failed pipeline jobs, do the following:

1. Navigate to the Logs Explorer:

   Go to the Logs Explorer

2. Select the appropriate Google Cloud project.

3. In the Resource drop-down menu, select Vertex Pipelines Job.

4. In the Location drop-down menu, select your pipeline's location.

5. In the All pipeline_job_id drop-down menu, select the ID of your pipeline job.

6. Click Apply.

7. Click Create Metric.

8. In the Create logs metric screen:

   1. Set the Metric type: Select Counter.

   2. Set the following fields in the Details section:

      • Log metric name: Enter a name for your log metric, for example, pipelinejob_failed. Choose a name that is unique among the logs-based metrics in your Google Cloud project. Some naming restrictions apply. For more information, see Troubleshooting.
      • Description: Enter a description for the metric.
      • Units: Leave this field blank or enter the digit 1.

   3. Define your metric filter in the Filter selection section. Create a filter that collects only the log entries that you want to count in your metric using the logging query language, for example:

      resource.type="aiplatform.googleapis.com/PipelineJob"
      jsonPayload.state="PIPELINE_STATE_FAILED"
      

      You can also use regular expressions to create your metric's filters.

      To open a panel showing you the log entries that match your filter, click Preview logs.

   4. (Optional) Add a label in the Labels section. For instructions on creating labels, see Create a label.

   5. To create the metric, click Create metric.

Create a custom metric for final pipeline state

To create a custom log-based counter metric that monitors the final state of your pipeline jobs, do the following:

1. Navigate to the Logs Explorer:

   Go to the Logs Explorer

2. Select the appropriate Google Cloud project.

3. In the Resource drop-down menu, select Vertex Pipelines Job.

4. In the Location drop-down menu, select your pipeline's location.

5. In the All pipeline_job_id drop-down menu, select the ID of your pipeline job.

6. Click Apply.

7. Click Create Metric.

8. In the Create logs metric screen:

   1. Set the Metric type: Select Counter.

   2. Set the following fields in the Details section:

      • Log metric name: Enter a name for your log metric, for example, Pipeline_state_final. Choose a name that is unique among the logs-based metrics in your Google Cloud project. Some naming restrictions apply. For more information, see Troubleshooting.
      • Description: Enter a description for the metric.
      • Units: Leave this field blank or enter the digit 1.

   3. Define your metric filter in the Filter selection section. Create a filter that collects only the log entries that you want to count in your metric using the logging query language, for example:

      resource.type="aiplatform.googleapis.com/PipelineJob"
      jsonPayload.state="PIPELINE_STATE_SUCCEEDED" OR
      "PIPELINE_STATE_FAILED" OR "PIPELINE_STATE_CANCELLED"
      

      You can also use regular expressions to create your metric's filters.

      To open a panel showing you the log entries that match your filter, click Preview logs.

   4. (Optional) Add a label in the Labels section. For instructions on creating labels, see Create a label.

   5. To create the metric, click Create metric.

View pipeline job failure rate

The pipeline job failure rate is calculated as the ratio of the number of pipeline jobs in the final state to the number of failed pipeline jobs. To create a dashboard to monitor the pipeline job failure rate, do the following:

1. Create a metric for monitoring the pipeline jobs in the final state. For more information, see Create a custom metric for final pipeline state.

2. Create a metric for monitoring the pipeline jobs in the failed state. For more information, see Create a custom metric for failed pipeline jobs.

3. In the Monitoring section of the Google Cloud console, go to the Metrics Explorer page.

   Go to the Metrics Explorer page

4. In the Configuration tab, do the following:

   1. Click Add another metric.

   2. Select the Display metrics as ratio option. When you select this option:

      • The Time series A pane is renamed Numerator.

      • The Time series B pane is renamed Denominator.

   3. In the Numerator pane, click Select a metric to select the metric created for failed pipeline jobs.

   4. In the Denominator pane, click Select a metric to select the metric created for final state pipeline jobs.

   5. (Optional) Configure the numerator and denominator by adding filters, or by updating the grouping fields and alignment parameters.

      For more information about how to add filters, or update the grouping fields and alignment parameters, see Chart a ratio of metrics.

      Note: The two metrics selected as the numerator and denominator must have the same resource type. If you select the same metric as the numerator and the denominator, use different filters or groupings to make the numerator distinct from the denominator.

   After you select the numerator and the denominator, the pipeline job failure rate is displayed in the graph.

   • After you generate the pipeline job failure rate graph, you can add it to your custom dashboard. For more information, see Save a chart for future reference.

   • To copy the URL containing the graph configuration, click the ellipsis icon on the top right corner of the graph, and then click Share by URL.

Route logs to a Cloud Pub/Sub sink

You can route pipeline logs to a Cloud Pub/Sub sink to build an event-driven architecture or to sync pipeline logs into your observability tools. For more information about routing logs to a Cloud Pub/Sub sink, see View logs routed to Pub/Sub.

This section shows you how to create a Pub/Sub topic, route (or sink) pipeline job logs to the topic, and verify the logs published to the topic. You can then subscribe downstream applications to this topic to power event-driven architectures, or sync pipeline logs with your observability tools in near real-time.

Console

To view logs routed to a Pub/Sub sink:

1. Create a Pub/Sub topic. For more information, see Create a topic in the Cloud Pub/Sub documentation.

2. On the Subscriptions page of the Google Cloud console, create a pull subscription for the Pub/Sub topic. For more information, see Create pull subscription.

3. Select Logging from the navigation menu, then click Go to Log Router:
   

   Go to Log Router

4. In your project containing the logs you want to route, click Create Sink.

5. Enter a name and description for the sink, and then click Next.

6. In the Service list, select Cloud Pub/Sub topic.

7. In the Destination list, select the Pub/Sub topic that you created, and then click Next.

8. Under Choose logs to include in sink, specify the following inclusion filter:

   resource.type="aiplatform.googleapis.com/PipelineJob"
   

9. Click Create Sink.

10. On the Subscriptions page, click the subscription ID.

    Go to Subscriptions page

11. Click the subcription ID that you created.

12. On the subscription details page, click the Messages tab.

13. To pull and view the pipeline job logs, click Pull.