Set up alerts and notifications for Dataplex tasks

This page describes how to set up alerts for failed job executions of a Dataplex task.

To set up alerts, you can use the Logs Explorer in the Google Cloud console or the Monitoring API.

Terminology

  • Incident autoclose duration: The duration after which an incident closes when there are no matching log entries..
  • Time between notifications: Minimum amount of time between receiving notifications for logs that match a filter. Any matching log within the minimum amount of time are skipped. The minimum time you can set is 5 minutes.

Set up notifications using Logs Explorer

  1. In the Google Cloud console, go to the Logs Explorer page.

    Go to Logs explorer

  2. Click the Show query toggle.

    1. To set up an alert for a failed job execution of any Dataplex task in a project, type the following query:

      severity=ERROR
resource.type = "dataplex.googleapis.com/Task"
logName = "projects/PROJECT_NAME/logs/dataplex.googleapis.com%2Fprocess"
jsonPayload.state = "FAILED"

    2. To set up an alert for a failed job execution for a specific Dataplex task in a project, type the following query:

      severity=ERROR
resource.type = "dataplex.googleapis.com/Task"
resource.labels.task_id = TASK_ID
logName = "projects/PROJECT_NAME/logs/dataplex.googleapis.com%2Fprocess"
jsonPayload.state = "FAILED"

  3. Click Create alert.

  4. In the Alert Policy Name field, enter a name for the new alert.

  5. In the Documentation field, enter the description you want to include in the notification. For more information about the content you can include in the documentation field of an alerting policy, see Using Markdown and variables in documentation templates.

  6. Click Next.

  7. Choose the logs you want to include in the alert, and click Next.

  8. Set the Time between notifications and the Incident autoclose duration, and click Next.

  9. Optional: Select a notification channel. For more information about configuring notification channels using the Google Cloud console, see Manage notification channels.

  10. Click Save.

Set up notifications using Monitoring API

  1. Optional: Enable the Monitoring API and authorize users to access it. For more information, see Enable the Monitoring API.
  2. Optional: Set the roles or permissions needed to create an alerting policy. For more information, see Authorization required to create alerting policies.
  3. Create a notification channel by either using Google Cloud console or Google Cloud CLI.

  4. Get the channel_id: 

    gcloud beta monitoring channels list

  5. Create a policy.json file with the following content for log-based-alerting-policy:

    1. Set up an alert for a failed job execution of any Dataplex task in a project:

      {
"displayName": "Alert Using Monitoring API",
"documentation": {
  "content": "Log-based alert in project PROJECT.",
  "mimeType": "text/markdown"
},

"conditions": [
  {
    "displayName": "Log match condition: Failed Job Execution",
    "conditionMatchedLog": {
      "filter": "severity=\"ERROR\" resource.type = \"dataplex.googleapis.com/Task\" logName = \"projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fprocess\" jsonPayload.state = \"FAILED\"",
    },
  }
],
"combiner": "OR",

"alertStrategy": {
  "notificationRateLimit": {
    "period": "300s"
  },
  "autoClose": "604800s",
},

"notificationChannels": [
  "projects/PROJECT_ID/notificationChannels/CHANNEL_ID"
]
}

    2. Set up an alert for a failed job execution for a specific Dataplex task in a project:

      {
"displayName": "Alert Using Monitoring API",
"documentation": {
  "content": "Log-based alert in project PROJECT.",
  "mimeType": "text/markdown"
},

"conditions": [
  {
    "displayName": "Log match condition: Failed Job Execution",
    "conditionMatchedLog": {
      "filter": "severity=\"ERROR\" resource.type = \"dataplex.googleapis.com/Task\" logName = \"projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fprocess\" jsonPayload.state = \"FAILED\" resource.labels.task_id = \"TASK_ID\"",
    },
  }
],
"combiner": "OR",

"alertStrategy": {
  "notificationRateLimit": {
    "period": "300s"
  },
  "autoClose": "604800s",
},

"notificationChannels": [
  "projects/PROJECT_ID/notificationChannels/CHANNEL_ID"
]
}

  6. Pass the policy.json file to the gcloud CLI:

    gcloud alpha monitoring policies create --policy-from-file="policy.json"

    If successful, the command returns the resource name of the new policy. For example:

    Created alert policy [projects/PROJECT_ID/alertPolicies/*POLICY_ID*].

What's next