Monitor Dataplex logs

Stay organized with collections Save and categorize content based on your preferences.

Dataplex job logs can be viewed, searched, filtered, and archived in Cloud Logging.

Access Dataplex service logs in Logging

Dataplex publishes the following service logs to Cloud Logging.

Log Type Log Name logName query Log Description
Discovery logs dataplex.googleapis.com/discovery logName=(projects/$PROJECT_ID/logs/dataplex.googleapis.com%2Fdiscovery) Discovery progress and updates over Assets in a Zone
Process logs dataplex.googleapis.com/process logName=(projects/$PROJECT_ID/logs/dataplex.googleapis.com%2Fprocess) Job runs resulting from data processing Tasks

Query discovery logs

When you use Dataplex to discover data in assets, a discovery log is produced in Logging.

You can access Logging using the Logging console, the gcloud logging command, or the Logging API.

Console

  1. In the Google Cloud console, go to the Cloud Logging > Logs (Logs Explorer) page:

    Go to the Logs Explorer page

  2. In the Logs Explorer view, find the Query tab.

  3. Click the Resource dropdown.

  4. Select Cloud Dataplex Zone. Click Add.

  5. Click the Log name dropdown.

  6. Type dataplex.googleapis.com%2Fdiscovery. Select discovery and click Add.

  7. (Optional) Filter the logs to a specific asset by adding the following filters in the log query:

    resource.labels.location="LOCATION"
    resource.labels.lake_id="LAKE_ID"
    resource.labels.zone_id="ZONE_ID"
    jsonPayload.assetId="ASSET_ID"
    
  8. Click Run query.

gcloud

The Google Cloud CLI has a group of commands, gcloud logging, that provide a command-line interface to the Cloud Logging API. To read your log entries, run the following command.

gcloud logging read \
    'resource.type="dataplex.googleapis.com/Zone" AND
    logName=projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fdiscovery AND
    resource.labels.location=LOCATION AND
    resource.labels.lake_id=LAKE_ID AND
    resource.labels.zone_id=ZONE_ID AND
    jsonPayload.assetId=ASSET_ID'
    --limit 10

REST

Follow the Logging API instructions to list log entries (entries.list) by using the APIs Explorer.

Query process logs

When you use Dataplex to schedule and run Tasks, a process log is produced in Logging for the resulting job.

You can access Logging using the Logging console, the gcloud logging command, or the Logging API.

Console

  1. In the Google Cloud console, go to the Cloud Logging > Logs (Logs Explorer) page:

    Go to the Logs Explorer page

  2. In the Logs Explorer view, find the Query tab.

  3. Click the Resource dropdown.

  4. Select Cloud Dataplex Task. Click Add.

  5. Click the Log name dropdown.

  6. Type dataplex.googleapis.com%2Fprocess. Select process and click Add.

  7. (Optional) Filter the logs to a specific task by adding the following filters in the log query:

    resource.labels.location="LOCATION"
    resource.labels.lake_id="LAKE_ID"
    resource.labels.task_id="TASK_ID"
    
  8. Click Run query.

gcloud

The Google Cloud SDK has a group of commands, gcloud logging, that provide a command-line interface to the Cloud Logging API. To read your log entries, run the following command.

gcloud logging read \
    'resource.type="dataplex.googleapis.com/Task" AND
    logName=projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fprocess AND
    resource.labels.location=LOCATION AND
    resource.labels.lake_id=LAKE_ID AND
    resource.labels.task_id=TASK_ID'
    --limit 10

REST

Follow the Logging API instructions to list log entries (entries.list) by using the APIs Explorer.

What's next